These are chat archives for ManageIQ/manageiq/performance

23rd
Nov 2015
Alex Krzos
@akrzos
Nov 23 2015 14:49
@kbrock managed to get satellite 6 refresh into my benchmarks now with pbench data on refreshes, you might be interested to view what the resource usage looks like
Keenan Brock
@kbrock
Nov 23 2015 14:49
thanks
Alex Krzos
@akrzos
Nov 23 2015 14:51
@dmetzger57 @jrafanie , I have some "long term" (8 Hours) memory usage graphs over 5.4.4.1 and 5.5.0.11 now as well, I'd like to share you guys have availability before the holiday
Keenan Brock
@kbrock
Nov 23 2015 14:52
sounds good
this week is getting crazy for me
wednesday is half day for the kids
tends not to be an issue
Alex Krzos
@akrzos
Nov 23 2015 14:53
maybe tomorrow works best to review rather than day before everyone gets off
Dennis Metzger
@dmetzger57
Nov 23 2015 14:54
tomorrow sounds good
Keenan Brock
@kbrock
Nov 23 2015 14:54
today would work better for me
brother in law may be swinging by - I may need to step away for a bit tomorrow
Alex Krzos
@akrzos
Nov 23 2015 14:56
I'll look at the calendar after my 10 o clock meeting to see what I can put together today or tomorrow
Keenan Brock
@kbrock
Nov 23 2015 15:00
that would be very cool - thanks
@akrzos "may be stepping away for a bit" ==> I'll be PTO
Alex Krzos
@akrzos
Nov 23 2015 15:04
@kbrock gotcha, no worries then I can share via email etc
Keenan Brock
@kbrock
Nov 23 2015 15:05
@akrzos I like your meetings. so if possible, today would be great
agree wednesday morning works for me - but not sure about others
wednesday PM probably not so good for most
Alex Krzos
@akrzos
Nov 23 2015 15:06
no one likes a "friday" meeting
Keenan Brock
@kbrock
Nov 23 2015 15:06
ya
Keenan Brock
@kbrock
Nov 23 2015 15:17
@matthewd how hard do you think it would be to write a cli for rack profiler ?
feel I'm kinda reimplementing that
to me, rack profiler looks like the client end of new relic rpm
wouldn't mind modifying the back end so we could send the results to our own data store as well
ooh - they have redis, file, memory - cool
Joe Rafaniello
@jrafanie
Nov 23 2015 15:22
@akrzos please do share, even if with just a basic background of what the graphs are showing (what was the performance scenario you're measuring, etc.) and anything you want to call out as "interesting"
Keenan Brock
@kbrock
Nov 23 2015 15:45
virtual columns are never going to work in large environments
and the solution is a bit invasive, but I'd imagine doable for most of the columns - at least the vms view (the most important one in my mine)
give the word if we want to do some big changes there
@dmetzger57 ^
Joe Rafaniello
@jrafanie
Nov 23 2015 15:49

virtual columns are never going to work in large environments

which part?

Keenan Brock
@kbrock
Nov 23 2015 15:50
@jrafanie the current definition of virtual columns (where the function is defined in ruby) will cause the server to download all data to the server, sort, and then throw away most of the data
Joe Rafaniello
@jrafanie
Nov 23 2015 15:50
I have a thing I found the other day where with_relationship_type was invalidating caches in memoist gem which means many caches in reporting appear to be less than usefull. I still need to get back to that
Keenan Brock
@kbrock
Nov 23 2015 15:50
many of these can be solved using an alternative implementation in sql
@jrafanie thanks - good to be reminded that there are other bugs lurking out there.
Joe Rafaniello
@jrafanie
Nov 23 2015 15:52
@kbrock I don't doubt that some things we can't do in ruby, but be wary of mass rewrites until we measure and find that we have reached the limits with the current architecture
no worries, keep in mind, much of this code was written before we had the ability to see allocation tracing or what's in the heap so it's very likely there are low hanging fruit we can find with more profiling
Dennis Metzger
@dmetzger57
Nov 23 2015 15:54
I’ve looked at a few “slow UI” issues recently and they all had use of virtual columns as a common theme. Need to quantifiy the impact of virtual columns to see if they are in fact an issue to be addressed or a red herring
Jason Frey
@Fryguy
Nov 23 2015 15:54
Were they virtual_columns? or were they bad usage of virtual_columns?
Dennis Metzger
@dmetzger57
Nov 23 2015 15:55
that is part of the “need to quantify"
Jason Frey
@Fryguy
Nov 23 2015 15:55
For example, if the :uses clause is not specified correctly...I would categorize that as the latter
Joe Rafaniello
@jrafanie
Nov 23 2015 15:55
@kbrock (for example, I was seeing enormous numbers of allocations in the memoist gem doing reporting locally... turns out it's because we're invalidating the cache badly for the reporting use case
Keenan Brock
@kbrock
Nov 23 2015 15:56
hmm. I guess my knee jerk is when we are sorting by a virtual (ruby) column. and we have thousands of rows
Keenan Brock
@kbrock
Nov 23 2015 16:06
I want a method log_if_takes_longer_than(1.second) { seed_database ; ... }
Dennis Metzger
@dmetzger57
Nov 23 2015 16:10
That could useful and expanded upon 😀
Keenan Brock
@kbrock
Nov 23 2015 16:11
forgot I had rack mini profiler installed on my box
on the login screen, we spend 1/2 second on render and no time in db.
darn - this seems like a rabit hole - may need to uninstall this gem ( :sparkles: )
layouts/exp_atom/_editor is a total BEAST
Alex Krzos
@akrzos
Nov 23 2015 17:35
So, just thought of this, not sure if it is solved in the code, but when a process swaps, its additional memory doesn't show up in RSS, since its not resident. My assumption is Miq does not take that additional memory usage into account, thus we could have a whole bunch of workers exceeding memory thresholds but because they exceed only when you account with swap, ManageIQ doesn't exit that worker. I would think more than likely things would timeout and the processes should eventually be replaced with new processes (With presumably lower memory usage)
Keenan Brock
@kbrock
Nov 23 2015 17:36
hmm
Alex Krzos
@akrzos
Nov 23 2015 17:36
Not sure if or how we could create that scenario, but just something to think about when using rss memory as a measurement
Keenan Brock
@kbrock
Nov 23 2015 17:36
yea
so, we would measure virtual memory instead?
will that ever go down?
Alex Krzos
@akrzos
Nov 23 2015 17:37
I don't think virtual memory is a great measurement for Miq, the processes definitely ask for more than they use
Keenan Brock
@kbrock
Nov 23 2015 17:37
yea
Alex Krzos
@akrzos
Nov 23 2015 17:37
PSS is another measurement I wish was easier to attain in RHEL
PSS vs RSS
Keenan Brock
@kbrock
Nov 23 2015 17:37
aah
Alex Krzos
@akrzos
Nov 23 2015 17:38
doesn't really affect our ruby workers that much since they don't seem to share "much" memory, but I need to make more measurements around it
but it definitely affects postgres
a running server with postgres with large shared buffers
shows more rss memory usage than all memory in the server, thus RSS has shared memory
The only scenario I can think of around the swap accounting issue would be someone creating more workers than "reasonable" and they consume all of the real memory and start consuming swap, but no workers break their memory threshold
Keenan Brock
@kbrock
Nov 23 2015 17:48
doesn't virtual memory requested affect swap? or is that really just a function of rss?
or if it is in swap really doesn't matter - - since it is swapped to disk and not read again?
Alex Krzos
@akrzos
Nov 23 2015 17:49
I'm not sure about that
I've seen Satellite 6 with tomcat have more than 40GiB of virtual memory but less than 1GiB of RSS and the machine had more than 100GiB of free memory
Alex Krzos
@akrzos
Nov 23 2015 17:55
So according to the advanced config, looks like there is some sort of swap usage kill that the worker monitor performs, that would limit the above scenario I mention I believe.
Keenan Brock
@kbrock
Nov 23 2015 18:11
would we get that kind of functionality if we didn't write our own worker runner?
does systemd or something already have that?
Alex Krzos
@akrzos
Nov 23 2015 18:14
vm.overcommit_ratio = 50 on 5.4 (RHEL6)
checking rhel7
same on rhel7 (5.5)
assuming centos with miq uses same values
Keenan Brock
@kbrock
Nov 23 2015 18:47

@Fryguy for vmware (17 vms):
capture_targets = capture_infra_targets + capture_cloud_targets + capture_container_targets
1.6mb = 1mb + 441kb + 35kb
I'm surprised that capture cloud and containers takes up memory

What was your idea about managers.flat_map?

Alex Krzos
@akrzos
Nov 23 2015 19:16
Would Wednesday 9am work on?
Keenan Brock
@kbrock
Nov 23 2015 19:16
This message was deleted
+1
akrzos @akrzos is looking at the calendar right now and that looks best
Dennis Metzger
@dmetzger57
Nov 23 2015 19:18
@akrzos Wednesday 9AM works
Keenan Brock
@kbrock
Nov 23 2015 19:31
9am-6PM mega meeting?
Alex Krzos
@akrzos
Nov 23 2015 19:31
uh oh
akrzos @akrzos checks
Alex Krzos
@akrzos
Nov 23 2015 19:31
9am to 10am
shows in my calendar
Keenan Brock
@kbrock
Nov 23 2015 19:31
phew