These are chat archives for ManageIQ/manageiq/performance

28th
Jan 2016
Joe Rafaniello
@jrafanie
Jan 28 2016 15:06
FYI, it looks like ruby 2.1/2.2/2.3 generational GC is not CoW friendly: https://twitter.com/tenderlove/status/692491754805788675 cc @akrzos
Keenan Brock
@kbrock
Jan 28 2016 15:07
that stinks
Joe Rafaniello
@jrafanie
Jan 28 2016 15:07
:pray: :cow:
Keenan Brock
@kbrock
Jan 28 2016 15:07
:heart: :cow:
Matthew Draper
@matthewd
Jan 28 2016 15:07
@jrafanie but a triple-GC before fork fixes everything, right?
Keenan Brock
@kbrock
Jan 28 2016 15:07
eat :chicken: :hamburger: instead
Joe Rafaniello
@jrafanie
Jan 28 2016 15:07
I'll be looking to see if I can promote our preloaded coded in the parent to be old objects helps
@matthewd yes, in a simple script, GC 3 times causes the memory to continue to be shared
it didn't work in the appliance I was testing before
I need to somehow test our app on 2.3.0 since it looks like the tri-color marking GC of 2.3.0 is a bit different than 2.2.x
Alex Krzos
@akrzos
Jan 28 2016 15:09
oh man
Joe Rafaniello
@jrafanie
Jan 28 2016 15:10
you're considered old if you survive 3 GCs
Daniel Berger
@djberg96
Jan 28 2016 15:10
that's some...odd code
Matthew Draper
@matthewd
Jan 28 2016 15:11
@djberg96 that's because it can't even
Joe Rafaniello
@jrafanie
Jan 28 2016 15:11
specifically, this is what @tenderlove and I were seeing in systemtap: https://twitter.com/jrafanie/status/692545012148600832
Daniel Berger
@djberg96
Jan 28 2016 15:11
I survived garbage collection 3 times and all I got was this lousy debug print statement.
Joe Rafaniello
@jrafanie
Jan 28 2016 15:12
LOL
be right back
Matthew Draper
@matthewd
Jan 28 2016 15:12
@jrafanie so in conclusion, when I said "something something systemtap?", I'd pretty much done 90% of the work for you.. right? :)
Joe Rafaniello
@jrafanie
Jan 28 2016 15:14
Lol, yes... And I had a belief that this was the case but systemtap confirms exactly why
Ruby 2.1p
Lol phones
Matthew Draper
@matthewd
Jan 28 2016 15:17
Yeah, I get the impression I should learn how to actually wield it at some point. It seems to have ended up giving you a very definitive answer.
Joe Rafaniello
@jrafanie
Jan 28 2016 15:17
Yeah, I can show anyone
Matthew Draper
@matthewd
Jan 28 2016 15:19
Anyway, hopefully t'lo can make it all better… though that doesn't seem likely to land before 2.4
So if we want something now, we probably need to find/prove a work-around (like the multi GC attack) that works on 2.2
.. given we don't know when appliances will be on 2.3
Joe Rafaniello
@jrafanie
Jan 28 2016 15:31
Yes, agreed
I need to see if I can run the workers with systemtap in the event the 3 times GC trick doesn't seem to "fix the shared -> private pages" in the child processes
Keenan Brock
@kbrock
Jan 28 2016 16:01
@matthewd verified it works on 2.2
leaning towards OS glob order - think you found it
Joe Rafaniello
@jrafanie
Jan 28 2016 16:01
wrong channel
Keenan Brock
@kbrock
Jan 28 2016 16:01
thanks
Joe Rafaniello
@jrafanie
Jan 28 2016 16:02
I got your back
;-)
Keenan Brock
@kbrock
Jan 28 2016 16:03
you sure do
Keenan Brock
@kbrock
Jan 28 2016 17:04
Is a virtual_column (with a uses clause) smart enough to auto includes() / sql join? Or do we still need the values in "include" in the product/views/*.yaml file?
To improve performance of the vm views, I'd like to get rid of these "include" which seem to be providing no benefit.
Jason Frey
@Fryguy
Jan 28 2016 17:12
virtual_column is not smart enough to have a default "uses"...I guess how could it?
but if you call .includes(some_virtual_column), it is smart enough to replace the includes with the uses
does that answer your question? I wasn't sure how to interpret it
Keenan Brock
@kbrock
Jan 28 2016 17:16
@Fryguy thanks it does
why would we need default uses?
never mind. I'll figure it out
thaks
Joe Rafaniello
@jrafanie
Jan 28 2016 21:10
ugh, so, the 3 times GC.start hack gets around the GC incrementing the age of each ruby object in it's header...
but it looks like a full GC mark walks all pages in the eden_heap and clears it's mark bits
leads to things like this in systemtap
__GI_memset+0xd2 [libc-2.17.so]
rgengc_mark_and_rememberset_clear+0x38 [ruby]
gc_marks_start+0xf8 [ruby]
gc_marks+0x4a [ruby]
gc_start+0x284 [ruby]
garbage_collect+0x40 [ruby]
gc_start_internal+0x1b3 [ruby]
call_cfunc_m1+0x2f [ruby]
vm_call_cfunc_with_frame+0x18d [ruby]
vm_call_cfunc+0x93 [ruby]
vm_call_method_each_type+0xbb [ruby]
vm_call_method+0x77 [ruby]
vm_call_general+0x3b [ruby]
vm_exec_core+0x24f0 [ruby]
vm_exec+0xe7 [ruby]
invoke_block+0xd6 [ruby]
invoke_block_from_c_0+0x206 [ruby]
even with https://github.com/ko1/nakayoshi_fork :crying_cat_face:
looks like those pages get copied
Keenan Brock
@kbrock
Jan 28 2016 21:13
@jrafanie are you thinking it is a bug with stuff we are doing, or a bug in ruby?
Joe Rafaniello
@jrafanie
Jan 28 2016 21:13
bug in ruby
I can't even get Koichi's readme script to report similar "benefits"
Daniel Berger
@djberg96
Jan 28 2016 21:16
posted on redmine?
Joe Rafaniello
@jrafanie
Jan 28 2016 21:18
Not yet, chatting with Aaron about where I should post my findings
I tried nakayoshi_fork on a nightly build... it has nearly identical USS(private), PSS(private + shared/N sharing pids)
it's slightly better in the USS and PSS column
Screen Shot 2016-01-28 at 12.47.55 PM.png
Screen Shot 2016-01-28 at 12.57.58 PM.png
Joe Rafaniello
@jrafanie
Jan 28 2016 21:23
The first is with nakayoshi_fork, the second is without it
Daniel Berger
@djberg96
Jan 28 2016 21:28
I'd post on redmine/ruby-core
Nick Carboni
@carbonin
Jan 28 2016 21:36
So I figure I can just drop these here - these are the initial results of BDR vs rubyrep performance testing.
ReplicationComparison.png
ReplicationDetail.png
This message was deleted
Just went ahead and build up a backlog of n rows and the let replication go.
Gist of the script I used to test and the results from no network throttling are here (https://gist.github.com/carbonin/1c5ac63ce304e2727e15)
Jason Frey
@Fryguy
Jan 28 2016 21:38
@jrafanie Are you saying you're surprised the bitmaps get copied? because I'm not...that's the point of the bitmaps
copying the bitmaps vs copying the entire Ruby object was the whole point of moving that information into the bitmap
Joe Rafaniello
@jrafanie
Jan 28 2016 21:39
I'm surprised at how much is being copied
I need to run systemtap with workers if it's possible
Jason Frey
@Fryguy
Jan 28 2016 21:40
@carbonin Those numbers are awesome :open_mouth:
Joe Rafaniello
@jrafanie
Jan 28 2016 21:40
@carbonin At first I thought you did the "Y axis doesn't start at 0" trick
those numbers are "it can't be right" numbers
Jason Frey
@Fryguy
Jan 28 2016 21:41
1 million rows in < 200s with 200ms ping is boggling my mind
maybe it shouldn't, but after rubyrep... :)
Nick Carboni
@carbonin
Jan 28 2016 21:41
Yeah, black magic :sparkles:
Joe Rafaniello
@jrafanie
Jan 28 2016 21:42
@carbonin are you sure it worked?
;-)
Nick Carboni
@carbonin
Jan 28 2016 21:42
Yup haha. But that's why I posted the gist, you can try for yourself :smile: Yay science :tada: