These are chat archives for ManageIQ/manageiq/performance

20th
Dec 2017
Nick LaMuro
@NickLaMuro
Dec 20 2017 00:24

ref: https://gitter.im/ManageIQ/manageiq/performance?at=5a397306ffa3e37919596034

So.... looks like I spoke too soon... and once again ran the first test wrong. Had to restart it about 20 min ago, and it seems to be rising steadily, but will keep looking at it.

Joe Rafaniello
@jrafanie
Dec 20 2017 02:05
Interesting @NickLaMuro
Keenan Brock
@kbrock
Dec 20 2017 02:06
whoa - what did you just say?
this is suggesting that our patches to drb may be an issue?
Joe Rafaniello
@jrafanie
Dec 20 2017 02:07
I think he needs to run the same exact test with and without the patches using the same test methodology
Keenan Brock
@kbrock
Dec 20 2017 02:08
think he had already run those tests w/o the patches to show nothing - but yea. maybe the test methodology changed - a different mythology or something
Joe Rafaniello
@jrafanie
Dec 20 2017 02:10
yeah, that's what I read into it too. The test was done differently. We'll have to wait and see what he finds.
Nick LaMuro
@NickLaMuro
Dec 20 2017 04:51
at this point, after letting it run (properly... by actually using bg and disown correctly...), I am still pretty skeptical that it is leaking
it didn't grow in "size" at all, and just shifted some memory around it seems
will check back every so often these next few days
Keenan Brock
@kbrock
Dec 20 2017 05:14
you know, you could always run using nohup or just plain & vs using control-z ; bg
Keenan Brock
@kbrock
Dec 20 2017 13:51
@NickLaMuro ok. So the all vms have 1 storage bug is frustrating me. Know it is a simulator only problem but... :(
Adam Grare
@agrare
Dec 20 2017 13:52
@kbrock what storage bug?
Keenan Brock
@kbrock
Dec 20 2017 14:44
@agrare simulators setup 1 storage to point to all VMs. so when you run a storage capture, it preloads every vm under a storage - bloating memory to be very large
Of course, in the real world, you don't have 1 storage assigned to 3k vms
Adam Grare
@agrare
Dec 20 2017 14:45
ah okay, you can configure the storage setup in the simulator config file
Keenan Brock
@kbrock
Dec 20 2017 14:45
this is a simulator only bug - but it does hit us often since most of the time QA/performance tests point to a simulated vmware
Adam Grare
@agrare
Dec 20 2017 14:45
i wonder if you gave it 0 global storages if it would put the vms on the host local datastores
Keenan Brock
@kbrock
Dec 20 2017 14:46
I haven't played around with that stuff too much
but it spikes memory to hell as it loads all vms into memory
Keenan Brock
@kbrock
Dec 20 2017 21:20
@NickLaMuro I've been sending my memory output to evm.log
but I'm beginning to wonder why
it is generating a unique line. and then I parse those values and generate a csv file
thinking I maybe should just produce a csv file to begin with
Nick LaMuro
@NickLaMuro
Dec 20 2017 21:42
honestly, I was just lazy and re-using a existing logger
Joe Rafaniello
@jrafanie
Dec 20 2017 21:59
So, I finally figured out the thread count for the server process. It seems to be:
main thread 
+ timer thread 
+ drb server thread 
+ (drb client + Process.detach resulting waiter thread) * number_of_workers
the waiter thread for each forked process is what I wasn't clear about
So, in other words, I'm hoping @NickLaMuro figures it out with this bisecting of code paths