These are chat archives for ManageIQ/manageiq/performance

19th
Jan 2018
Keenan Brock
@kbrock
Jan 19 2018 00:17
:(
@NickLaMuro what moron added that path to the auth_config_model_spec.rb? :(
though, went from 2 files changed/added in that PR to 17 files...
Keenan Brock
@kbrock
Jan 19 2018 00:19
2... 17... what is an extra 7500% 750% between friends?
Nick LaMuro
@NickLaMuro
Jan 19 2018 00:21
uh... that math seems twice 10x as big as it should be...
750% ?
Keenan Brock
@kbrock
Jan 19 2018 00:22
lol, I was laughing at the twice too
Nick LaMuro
@NickLaMuro
Jan 19 2018 00:22
stop cheating @kbrock ...
Keenan Brock
@kbrock
Jan 19 2018 00:22
fixed
Keenan Brock
@kbrock
Jan 19 2018 14:29
wish scp remote:/x.gz would gzip the file if only remote:/x existed
Keenan Brock
@kbrock
Jan 19 2018 14:36
evm_MetricsCollectorWorker_pid_29797.png
Patched an appliance with the to_s. NOP'd storage - after a day, looks like the metrics collector has stabalized
evm_MetricsCollectorWorker_pid_29797-9.png
zoomed into the last hour shows a nice consistency. the GC every 15 minutes (default behavior) is nice as well
will let run for just a little more
Keenan Brock
@kbrock
Jan 19 2018 14:41
thinking of enabling storage to see if it spikes the env to hell (simulator have a single storage for every VM - and there are 1k)
ok - this has been running for less than a day - so this is only preliminary
@dmetzger57 / @NickLaMuro any plans to manually patch the appliances and see how this goes in the longer term?
I used:
grep -r autoload_paths lib config app /opt/rh/cfme-gemset/bundler/gems/*/lib
Dennis Metzger
@dmetzger57
Jan 19 2018 14:46
@kbrock I setup 5.8 and 5.9 appliances for that, @NickLaMuro was patching
patching with the "official" proposed patch
Nick LaMuro
@NickLaMuro
Jan 19 2018 15:34
@kbrock yes, this is done, and I will be posting some of the results this morning
I wanted to run it first and have it leak, so we had a baseline, so the first day was just getting that
Dennis Metzger
@dmetzger57
Jan 19 2018 15:36
:+1:
Keenan Brock
@kbrock
Jan 19 2018 15:37
:+1: I jumped the gun and am very excited about the results. especially with the Metrics Collector
Joe Rafaniello
@jrafanie
Jan 19 2018 16:27
@NickLaMuro congrats, your PRs to workaround the memory leak are all merged :tada:
nice work!
Chris Arcand
@chrisarcand
Jan 19 2018 16:28
:clap:
Joe Rafaniello
@jrafanie
Jan 19 2018 16:28
@kbrock did you see my mention of instruments on osx?
in case you can run the leaking worker via rake evm:start on osx, you can use instruments to record the process and it will tell you if it's leaking
apparently, @Fryguy knew about this and didn't tell us how awesome and simple it is :angry: :laughing: :rofl:
Keenan Brock
@kbrock
Jan 19 2018 16:54
@jrafanie I started using instruments yesterday when I was trying to ensure that the bug was with pathname in require and not with the strings not matching or freeze/intern a string
I tend to avoid evm:start on my mac
we need to up our tracing game
Nick LaMuro
@NickLaMuro
Jan 19 2018 17:46
currently, we probably aren't as "microservice-y" as they are
Keenan Brock
@kbrock
Jan 19 2018 18:06
true
but ui actions do trigger miq messages and background work
latest tool describes a log file (or multiple)
describe_log.rb --group --details --messages evm.log-201801*

ManageIQ::Providers::Vmware::InfraManager::MetricsCollectorWorker (1)
     29797 [1117739] 2018-01-18T15:41:47 -- 2018-01-19T09:27:48
           [ 200247] Storage.perf_capture_historical
           [  21491] Storage.perf_capture_hourly
           [  12734] ManageIQ::Providers::Vmware::InfraManager::Vm.perf_capture_realtime
           [   2073] ManageIQ::Providers::Vmware::InfraManager::Vm.perf_capture_historical
           [   1325] ManageIQ::Providers::Vmware::InfraManager::HostEsx.perf_capture_realtime
           [     45] ManageIQ::Providers::Vmware::InfraManager::HostEsx.perf_capture_historical
MiqEmsMetricsProcessorWorker (1)
     29807 [ 136675] 2018-01-18T15:41:47 -- 2018-01-19T09:16:39
           [    441] EmsCluster.perf_rollup_range
           [    397] EmsCluster.perf_rollup
           [     48] MiqRegion.perf_rollup
...
don't quite cross reference top logs with evm.log in graphs yet, but getting there
Nick LaMuro
@NickLaMuro
Jan 19 2018 18:12
I am working on some stuff as well. Hope to have it pushed up later today (will try to remember to make it a PR instead of a push to master as well)
Keenan Brock
@kbrock
Jan 19 2018 18:12
meh
just put it
I'm merging my job info (bash) into the describe_log.rb
will probably have a describe_queue
the last BZ - with the recycled PIDs really got me diving deep into describe_log - there were hundreds (if not thousands) of pids
feel like the tools should output more of a csv output, and then a generic grouping/ formatter to get it the way we want
would be cool to say group_by=1,2,3,display=4,count=5 (in essence what I did above)
Keenan Brock
@kbrock
Jan 19 2018 18:24
Nick LaMuro
@NickLaMuro
Jan 19 2018 18:29

@NickLaMuro I plagiarized you

Good. It's DRY.

Keenan Brock
@kbrock
Jan 19 2018 18:30
Wish it were the solution to the GenericWorker - but I'm not sold that the fix delivers for that issue.
(pretty confident the metrics collector is fixed)
Nick LaMuro
@NickLaMuro
Jan 19 2018 18:30

feel like the tools should output more of a csv output

Yeah, that is a good point. Pretty sure gnuplot can also read from a CSV input, I just haven't dug in enough to get that working properly

Keenan Brock
@kbrock
Jan 19 2018 18:31
or space delimited or something
just noticed in my tooling I'm creating very basic pivot capability

instead of outputting

PID WorkerType Message

I output

WorkerType
  PID
  PID
    Message #message_count
would be cool if we coud output the first syntax and convert it into the second
Nick LaMuro
@NickLaMuro
Jan 19 2018 18:33
For what it is worth, what I am currently working on is abstracting the regexp matching/pid stuff to be something more... "shareable"?
Keenan Brock
@kbrock
Jan 19 2018 18:33
yea
with the split_log stuff, I output a data file (like you requested)- I was looking at tweaking the regex so you could control what ended up in the data file
Nick LaMuro
@NickLaMuro
Jan 19 2018 18:34
This could do that
Keenan Brock
@kbrock
Jan 19 2018 18:34
of course it can
it can slice and dice and ...
Nick LaMuro
@NickLaMuro
Jan 19 2018 18:34
going to be converting the regexp stuff to use named_captures, which will remove a lot of the boilerplate code
Keenan Brock
@kbrock
Jan 19 2018 18:35
nice
they make for ugly regex, but very cool in theory
Nick LaMuro
@NickLaMuro
Jan 19 2018 18:35
but that is also why I want your opinion on the final result, because it is going to be a huge refactor
Keenan Brock
@kbrock
Jan 19 2018 18:35
yea
but you and I've re-written that parser ~5+ times at least
so it will be good to consolidate / standardize
at first, I'd like thought about changing the top processor to just put the date inline - so it is a flat file, then we could slice and dice how we need
then we could split it or group it or filter it or ...
Dennis Metzger
@dmetzger57
Jan 19 2018 18:38
@NickLaMuro named_captures :+1: :clap:
Keenan Brock
@kbrock
Jan 19 2018 18:38
lol
:hammer: :nail: :) -- if you have a :hammer: then the world is made up of nails
Dennis Metzger
@dmetzger57
Jan 19 2018 18:41
if you have a good screw gun the world is made up screws, get with the times
Keenan Brock
@kbrock
Jan 19 2018 19:08
no emoji for :screw gun:
there is a pun in there somewhere. probably more than one
Jason Frey
@Fryguy
Jan 19 2018 19:10
not sure it is what you are referring to @NickLaMuro but I feel obligated to share this with you if it helps: https://github.com/ManageIQ/manageiq-gems-pending/blob/master/lib/gems/pending/util/miq_logger_processor.rb
Nick LaMuro
@NickLaMuro
Jan 19 2018 19:11
I think I have known about it, but definitely should revisit the code for it again
Jason Frey
@Fryguy
Jan 19 2018 19:12
Keenan Brock
@kbrock
Jan 19 2018 20:24
as I'm looking at this more and more, I'm feeling like we need to emit AS:Notifications and have our logging scripts decide what to capture
the once sized fits all logging we are doing in stuff like server monitor isn't helping me understand the profile
also requires too much monkey patching to change it much
Nick LaMuro
@NickLaMuro
Jan 19 2018 20:26
You've said this a lot, has anyone told you "no" yet?
Keenan Brock
@kbrock
Jan 19 2018 20:26

STFU and do it already

lol

(aka thanks)
Nick LaMuro
@NickLaMuro
Jan 19 2018 20:28
well, it was a legit question to. If someone has raise an actual concern/dislike for the idea, then yeah, we should talk it out more before you invest some time in it. But if not, toss up a POC and we can then discuss it with more concrete stuff around it.
Keenan Brock
@kbrock
Jan 19 2018 20:42
yea, on my appliances, I'm basically monkeypatching to add them
Oleg Barenboim
@chessbyte
Jan 19 2018 20:44
@NickLaMuro if all the PRs for https://bugzilla.redhat.com/show_bug.cgi?id=1535720 are merged, you should move it to POST - so that @simaishi can start backporting
Nick LaMuro
@NickLaMuro
Jan 19 2018 20:45
@chessbyte sorry, got that mixed up. will do that now (thought I had to do that after backport had happened)
done
Keenan Brock
@kbrock
Jan 19 2018 20:49
nick - that was such a good fix. thanks a lot (kudos to @jrafanie and others too)
Joe Rafaniello
@jrafanie
Jan 19 2018 20:51
:+1:
Hopefully, we'll forget all about our memory problems ;-)
Nick LaMuro
@NickLaMuro
Jan 19 2018 20:57
You just couldn’t resist huh...
:expressionless:
Keenan Brock
@kbrock
Jan 19 2018 20:57
if we forget about it, won't it be a memory problem?
a problem remembering our memory problems?
Joe Rafaniello
@jrafanie
Jan 19 2018 20:58

You just couldn’t resist huh...

Nope

Keenan Brock
@kbrock
Jan 19 2018 21:37
@NickLaMuro deliminate the act of eliminating delimiters
@NickLaMuro re the changes - it continuously displays output. it uses "\r" to rewrite the line with the updates
Keenan Brock
@kbrock
Jan 19 2018 21:42
hmm, I'll comment inline
Keenan Brock
@kbrock
Jan 19 2018 22:32
@NickLaMuro hmm... I find myself splitting hairs to try and get rid of lines of code like puts. And from what I know about you, this doesn't make sense to me.
Was your issue because you didn't think it was updating the screen in realtime? (If that were the case, then yes, I agree with you - this would be a stupid PR)
Nick LaMuro
@NickLaMuro
Jan 19 2018 22:34
\r's are something that make little sense when piping into a file
Keenan Brock
@kbrock
Jan 19 2018 22:35
and you pipe this to a file?
Nick LaMuro
@NickLaMuro
Jan 19 2018 22:35
also, I commented in the PR already that basically would be my response to what you just said
Keenan Brock
@kbrock
Jan 19 2018 22:35
thnx
Nick LaMuro
@NickLaMuro
Jan 19 2018 22:35

and you pipe this to a file?

Yeah, if I want to set this to run over night

Keenan Brock
@kbrock
Jan 19 2018 22:36
cool - sounds like a plan
Nick LaMuro
@NickLaMuro
Jan 19 2018 22:36
why didn't you just keep the PR open?
Keenan Brock
@kbrock
Jan 19 2018 22:36
what is the point?
the original script is a simple memory profiling
Nick LaMuro
@NickLaMuro
Jan 19 2018 22:36
The discussion is all in one spot?