These are chat archives for ManageIQ/manageiq/performance

13th
Oct 2017
Keenan Brock
@kbrock
Oct 13 2017 13:43
@Fryguy @jrafanie is it possible to turn off "use drb" for a worker? I want to see if it leaks when I stop using drb
Jason Frey
@Fryguy
Oct 13 2017 13:44
no, I don't think so
Joe Rafaniello
@jrafanie
Oct 13 2017 13:54
Keenan Brock
@kbrock
Oct 13 2017 13:55
@jrafanie so I run generic worker via run_single_worker.rb - what is -b?
Joe Rafaniello
@jrafanie
Oct 13 2017 13:55
It might not work for everything but ultimately, that's the direction we want to go so it's better to find and fix those areas now
see link, don't heartbeat
Keenan Brock
@kbrock
Oct 13 2017 13:55
thnx
Keenan Brock
@kbrock
Oct 13 2017 19:29
@dmetzger57 the longer term graphs
plot-30.png
hard to see the pattern in there, but if you zoom in for ~1 day
Keenan Brock
@kbrock
Oct 13 2017 19:36
plot-0-26hrs.png
the reset of the cache (number of old objects) looks more prevalent,
and the vertical blue (number of new objects) is more obvious
still not sure why the sudden jump in RSS/PSS @ 2 hours.
maybe that is how much time GC needed to completely replace the old objects. maybe that is finally when all the old objects were finally disposed, and a bunch of pages were cleared or something
Keenan Brock
@kbrock
Oct 13 2017 19:42
fun: in my datafile, I have every MiqQueue message linked to each data points, so I can tell you which message occured at each value in the graph
Keenan Brock
@kbrock
Oct 13 2017 21:21
ok, I'm fried. what I've learned
  1. cache_with_timeout: any method protected by cache_with_timeout blocks all other methods defined by cache_with_timeout - discussed in core room. holding off
  2. some methods like VmdbDatabase.capture_metrics_timer simply take 3 minutes to run (and run every hour) - this explains some of the gaps
  3. something is caching 1.5M objects. it gets cleared out at 15:00 (T+3.5), 20:00 (T+8:00), 2:00 (T+16.5), 6:30. IT doesn't take very long to load all these objects back into memory TODO: I need to track down each method call into these. it seems random so far
Jason Frey
@Fryguy
Oct 13 2017 21:23
where does VmdbDatabase.capture_metrics_timer collide with cache_with_timeout?
are those related at all?