These are chat archives for ManageIQ/manageiq/performance

7th
Mar 2017
Dennis Metzger
@dmetzger57
Mar 07 2017 15:44 UTC

Completed a test looking at PSS memory usage on CFME 5.8.0.2-1, during provider add and refresh

https://docs.google.com/a/redhat.com/spreadsheets/d/1m9R16vBQmuiv0jheutwxUZ1oXUW7hiNaF3wJRnagtCY/edit?usp=sharing

Looks like we've regressed in PSS usagae in a number of worker types

Will correlate with QE results which were updated last night

In depth looks at the workers with the biggest negative impact in the future

A run just completed which include CnU and Proxy workers, will post when I collect / validate the results

Jason Frey
@Fryguy
Mar 07 2017 15:46 UTC
Thanks @dmetzger57
Looks like a consistent bump for the entirety of the graph
Dennis Metzger
@dmetzger57
Mar 07 2017 15:49 UTC
defintely a “base” bump in there
Jason Frey
@Fryguy
Mar 07 2017 15:49 UTC
Yeah
Joe Rafaniello
@jrafanie
Mar 07 2017 15:50 UTC
yeah, is this is the first time we're measuring 5.8?
Jason Frey
@Fryguy
Mar 07 2017 15:50 UTC
I have to work with @jrafanie on a different bug, so maybe we can peek at it
Joe Rafaniello
@jrafanie
Mar 07 2017 15:51 UTC
In the PSS deltas, it looks like we looked at 5.6 to 5.7 but didn't keep up with 5.8
did any worker exceed their memory threshold and get restarted?
Keenan Brock
@kbrock
Mar 07 2017 15:51 UTC
reporting
I'm guessing
but you can see a major dip there
Joe Rafaniello
@jrafanie
Mar 07 2017 15:52 UTC
yeah, I see what you mean @kbrock , thanks
Dennis Metzger
@dmetzger57
Mar 07 2017 15:52 UTC
no restarts, if they did they retained PIDs
Joe Rafaniello
@jrafanie
Mar 07 2017 15:52 UTC
ok
Dennis Metzger
@dmetzger57
Mar 07 2017 15:52 UTC
which would amaze me
Keenan Brock
@kbrock
Mar 07 2017 15:52 UTC
huh - yea, what dennis said ;)
Dennis Metzger
@dmetzger57
Mar 07 2017 15:53 UTC
i have the associated logs, will check there too
Joe Rafaniello
@jrafanie
Mar 07 2017 15:54 UTC
I wonder how we can stay ahead of this problem since we'll always be adding things, not removing them
Dennis Metzger
@dmetzger57
Mar 07 2017 15:56 UTC
well, now that I’ve scripted this I’ll be running my own tests, plus I’ve requested again runs against downstream builds (at least weekly) from QE (Idle Memory) which they’ve agreed to start
now, for upstream
Joe Rafaniello
@jrafanie
Mar 07 2017 15:58 UTC
Does it make sense to have QE run with 10%-20% reduced memory threshold limits and open bugs if we raise any evm_worker_memory_exceeded events?
Keenan Brock
@kbrock
Mar 07 2017 15:59 UTC
I guess. as long as it doesn't raise too many events.
they kinda just want to get their job done.
Joe Rafaniello
@jrafanie
Mar 07 2017 15:59 UTC
to be honest, 10-20% is only 50 MB PSS for some workers and is not much "room" for growth
Keenan Brock
@kbrock
Mar 07 2017 15:59 UTC
aren't our numbers tuned to be close to the mark?
Joe Rafaniello
@jrafanie
Mar 07 2017 16:00 UTC
I think tuned is the wrong word
Keenan Brock
@kbrock
Mar 07 2017 16:00 UTC
s/tuned/guessed/?
it is "tuned" to run at 5 mpg?
Dennis Metzger
@dmetzger57
Mar 07 2017 16:00 UTC
we can tell if a worker would have exceeded, the max values was being added (have not yet looked at yesterday results) to the QE test results