These are chat archives for ManageIQ/manageiq/performance

17th
Sep 2015
Keenan Brock
@kbrock
Sep 17 2015 18:04
@akrzos collectd reading postgres:
query to fetch statistics from postgres borrowed from sensu
collectd plugin to run queries on postgres plugin
think you can get postgres data into influx in < 30 minutes
Alex Krzos
@akrzos
Sep 17 2015 18:07
@kbrock I'll give it a go
Keenan Brock
@kbrock
Sep 17 2015 18:08
I don't know collectd, nor how to install plugins... but that looked like the cheapest way to get it
Matthew Draper
@matthewd
Sep 17 2015 18:09
I think I used to pull PG stats into collects via a munin compatibility thing
But that was probably only because I was using munin before that
Alex Krzos
@akrzos
Sep 17 2015 18:42
Is that a GC pause for 4 seconds? https://gist.github.com/akrzos/a09296f5fe7fb11bf061
Would be sweet to have collectd tell me
Keenan Brock
@kbrock
Sep 17 2015 18:44
@akrzos I still need some time before I can fully dive into here, but I really want to solve that. so when we're improving this suite, you'll have even better graphs to show off your metrics_fu
Alex Krzos
@akrzos
Sep 17 2015 18:44
@kbrock no worries, just saw that and I'm pretty sure that a GC pause
happening to be tailing vim.log
and was seeing that
Joe Rafaniello
@jrafanie
Sep 17 2015 18:45
@akrzos can’t you change our logger formatter to throw in GC.count on each log line?
Alex Krzos
@akrzos
Sep 17 2015 18:45
@jrafanie interesting
hmm, then we can have per pid gc.count
@akrzos some of those GC values may require you to turn on the profiler
GC.count works for me in irb so it’s probably the timing and object count/sizes that require the profiler
YMMV
Matthew Draper
@matthewd
Sep 17 2015 18:49
Would it be easier to just log the actual GC events? https://github.com/ko1/gc_tracer
Well, maybe s/easier/more informative/ ¯\_(ツ)_/¯
Joe Rafaniello
@jrafanie
Sep 17 2015 18:51
is that something we can run in production @matthewd? how much overhead is it?
GC.count and basic stuff I would assume has very little to no overhead
typing is not my friend today
Matthew Draper
@matthewd
Sep 17 2015 18:52
Given that it (AIUI) does some stuff at GC time [i.e., rarely], I don't imagine it would have a huge overhead
Keenan Brock
@kbrock
Sep 17 2015 18:52
I added a file in config/initializers/ that changes my logging output. let me know if you need help
Matthew Draper
@matthewd
Sep 17 2015 18:53
I mean, I don't think I'd just turn it on everywhere all the time, just for the sake of it… but to track things in a perf environment, I'd default to assuming it'll work, I guess
Joe Rafaniello
@jrafanie
Sep 17 2015 18:54
Cool, @akrzos please kick the tires and report back here. ruby 2.1/2.2’s GC and tooling will get us easy wins
Matthew Draper
@matthewd
Sep 17 2015 18:54
I mean, it's a tool for making sure GC isn't badly affecting your perf, so surely if it killed things too badly, it'd defeat itself
Joe Rafaniello
@jrafanie
Sep 17 2015 18:54
right
Alex Krzos
@akrzos
Sep 17 2015 18:54
@jrafanie I'll see what I can whipe up around GC.stat in the logger as well
Joe Rafaniello
@jrafanie
Sep 17 2015 18:55
@akrzos what logger were you looking at? vim.log?
Alex Krzos
@akrzos
Sep 17 2015 18:55
vim.log
Matthew Draper
@matthewd
Sep 17 2015 18:55
That said, I didn't think 2.2's GC would pause for anywhere near that long
Alex Krzos
@akrzos
Sep 17 2015 18:55
thats from a 5.4.2.0 appliance thus Ruby 2.0
Matthew Draper
@matthewd
Sep 17 2015 18:56
(because avoiding the occasional long stops of 2.1's RGenGC was the primary aim of 2.2's RIncGC)
Joe Rafaniello
@jrafanie
Sep 17 2015 18:57
@akrzos some of the basic GC stuff shoudl work with ruby 2.0 http://ruby-doc.org/core-2.0.0/GC.html#method-c-count
And vim is a MirroredLogger which is a subclass of VMDBLogger so you shoudl be able to modify that formatter like the link I sent you
even just modify that in place to add your stats
Alex Krzos
@akrzos
Sep 17 2015 19:43
4s that time
and you can see my GC counter
thats nuts
Matthew Draper
@matthewd
Sep 17 2015 19:45
Yeah, that's why GC improvements were amongst the headline features of both 2.1 and 2.2 :grinning:
Alex Krzos
@akrzos
Sep 17 2015 19:45
I should put the same scenario in play on the 5.5 alpha and see the difference
Matthew Draper
@matthewd
Sep 17 2015 19:46
.. though the 2.2 improvements were about fixing the fact that 2.1 could have 15ms pauses
Joe Rafaniello
@jrafanie
Sep 17 2015 19:55
@akrzos how big is the process at that time? That’s ruby 2.0, right?
Alex Krzos
@akrzos
Sep 17 2015 19:56
@jrafanie let me dig up it's size before hand though all i can dig its size from would be miqtop on this appliance so closest 1m interval i can find to it
Joe Rafaniello
@jrafanie
Sep 17 2015 19:56
yeah, ballpark figure is good enough and if you’re out of resident memory… swapping
Alex Krzos
@akrzos
Sep 17 2015 19:57
oh shouldn't be swapping, i bump any appliance that connects to xlarge providers to 16GiB
Joe Rafaniello
@jrafanie
Sep 17 2015 19:57
vim broker can probably be really improved by freezing strings on ruby 2.2 so they share the same string object
the vim mapping registry is pretty big and most of it are strings that are probably duplicated alot
Alex Krzos
@akrzos
Sep 17 2015 20:01
hmm I think top output wasn't running at that point in time
anyway the process is 3.1g in rss size right now
3264132 kB
to be more exact
Joe Rafaniello
@jrafanie
Sep 17 2015 20:03
that would explain 4 second GC pause