These are chat archives for ManageIQ/manageiq/performance

7th
Mar 2017
Dennis Metzger
@dmetzger57
Mar 07 2017 15:44

Completed a test looking at PSS memory usage on CFME 5.8.0.2-1, during provider add and refresh

https://docs.google.com/a/redhat.com/spreadsheets/d/1m9R16vBQmuiv0jheutwxUZ1oXUW7hiNaF3wJRnagtCY/edit?usp=sharing

Looks like we've regressed in PSS usagae in a number of worker types

Will correlate with QE results which were updated last night

In depth looks at the workers with the biggest negative impact in the future

A run just completed which include CnU and Proxy workers, will post when I collect / validate the results

Jason Frey
@Fryguy
Mar 07 2017 15:46
Thanks @dmetzger57
Looks like a consistent bump for the entirety of the graph
Dennis Metzger
@dmetzger57
Mar 07 2017 15:49
defintely a “base” bump in there
Jason Frey
@Fryguy
Mar 07 2017 15:49
Yeah
Joe Rafaniello
@jrafanie
Mar 07 2017 15:50
yeah, is this is the first time we're measuring 5.8?
Jason Frey
@Fryguy
Mar 07 2017 15:50
I have to work with @jrafanie on a different bug, so maybe we can peek at it
Joe Rafaniello
@jrafanie
Mar 07 2017 15:51
In the PSS deltas, it looks like we looked at 5.6 to 5.7 but didn't keep up with 5.8
did any worker exceed their memory threshold and get restarted?
Keenan Brock
@kbrock
Mar 07 2017 15:51
reporting
I'm guessing
but you can see a major dip there
Joe Rafaniello
@jrafanie
Mar 07 2017 15:52
yeah, I see what you mean @kbrock , thanks
Dennis Metzger
@dmetzger57
Mar 07 2017 15:52
no restarts, if they did they retained PIDs
Joe Rafaniello
@jrafanie
Mar 07 2017 15:52
ok
Dennis Metzger
@dmetzger57
Mar 07 2017 15:52
which would amaze me
Keenan Brock
@kbrock
Mar 07 2017 15:52
huh - yea, what dennis said ;)
Dennis Metzger
@dmetzger57
Mar 07 2017 15:53
i have the associated logs, will check there too
Joe Rafaniello
@jrafanie
Mar 07 2017 15:54
I wonder how we can stay ahead of this problem since we'll always be adding things, not removing them
Dennis Metzger
@dmetzger57
Mar 07 2017 15:56
well, now that I’ve scripted this I’ll be running my own tests, plus I’ve requested again runs against downstream builds (at least weekly) from QE (Idle Memory) which they’ve agreed to start
now, for upstream
Joe Rafaniello
@jrafanie
Mar 07 2017 15:58
Does it make sense to have QE run with 10%-20% reduced memory threshold limits and open bugs if we raise any evm_worker_memory_exceeded events?
Keenan Brock
@kbrock
Mar 07 2017 15:59
I guess. as long as it doesn't raise too many events.
they kinda just want to get their job done.
Joe Rafaniello
@jrafanie
Mar 07 2017 15:59
to be honest, 10-20% is only 50 MB PSS for some workers and is not much "room" for growth
Keenan Brock
@kbrock
Mar 07 2017 15:59
aren't our numbers tuned to be close to the mark?
Joe Rafaniello
@jrafanie
Mar 07 2017 16:00
I think tuned is the wrong word
Keenan Brock
@kbrock
Mar 07 2017 16:00
s/tuned/guessed/?
it is "tuned" to run at 5 mpg?
Dennis Metzger
@dmetzger57
Mar 07 2017 16:00
we can tell if a worker would have exceeded, the max values was being added (have not yet looked at yesterday results) to the QE test results