These are chat archives for ManageIQ/manageiq/performance

31st
May 2016
Alex Krzos
@akrzos
May 31 2016 12:03
So guys I got a working example of MiqQueue dumping dequeue and deliver timings into statsd and then into my Graphite TSDB
Oleg Barenboim
@chessbyte
May 31 2016 13:37
@akrzos very cool
Keenan Brock
@kbrock
May 31 2016 13:57
@akrzos very cool. and sorry I didn't get there sooner
Alex Krzos
@akrzos
May 31 2016 13:58
@kbrock I can use your help with the namespacing of metrics I can't seem to get this right plus things like this are throwing me off : "ManageIQ::Providers::Vmware::InfraManager::Vm.perf_capture_realtime" All those double colons are causing messages to be namespaced further
If you want to poke around at my sandbox for it please have at it
Joe Rafaniello
@jrafanie
May 31 2016 14:02
@akrzos How easy is it for you to spin up a 5.6 test at the same level of an existing test?
Alex Krzos
@akrzos
May 31 2016 14:03
Maybe takes about 5 minutes for me to spin up a monitored 5.6 appliance
Joe Rafaniello
@jrafanie
May 31 2016 14:03
The lazy loaded message catalogs just merged upstream
I don't know how much benefit they'll have
an easy way to test it to remove the non-english message catalogs on an existing 5.6 appliance
.//config/locales/ja/manageiq.po
.//config/locales/nl/manageiq.po
.//config/locales/zh_CN/manageiq.po
Alex Krzos
@akrzos
May 31 2016 14:05
Gotcha, and we would like to see the worker memory differences with that change I assume right?
Joe Rafaniello
@jrafanie
May 31 2016 14:05
local testing shows loading these additional catalogs can take up to 15 MB per process
Yes, the CoW friendliness of ruby makes this hard to predict
Alex Krzos
@akrzos
May 31 2016 14:05
so just remove those files or is this a patch?
Joe Rafaniello
@jrafanie
May 31 2016 14:06
Just remove those files in the testing appliance... again, if it's easy
otherwise, i can get manageiq merged and wait for the next build
Alex Krzos
@akrzos
May 31 2016 14:06
won't be hard at all
I normally let it idle for an hour incase some other background things decided to use up more memory
Joe Rafaniello
@jrafanie
May 31 2016 14:07
Great, thanks!
Keep in mind though that even the english catalog is much larger now than it was on 5.5
For reference, I did a comparison of what's in the heap after rails boot comparison of 5.5.4.2 and 5.6.0.7
5.5.4.2
m:     961,119 | c:   9,507 | String  /Users/joerafaniello/.rubies/ruby-2.2.5/lib/ruby/2.2.0/racc/parser.rb:258
m:     613,785 | c:  10,013 | String  poparser.ry:139
m:     570,334 | c:  10,013 | String  poparser.ry:219
m:     499,920 | c:       4 | FastGettext::GetText::MOFile  /Users/joerafaniello/.gem/ruby/2.2.5/gems/fast_gettext-1.0.0/lib/fast_gettext/po_file.rb:14
m is memory size as ruby sees it
c is count of objects of that type from that location
5.6.0.7
m:   1,123,649 | c:  10,749 | String  /Users/joerafaniello/.rubies/ruby-2.2.5/lib/ruby/2.2.0/racc/parser.rb:258
m:   1,418,546 | c:  20,182 | String  poparser.ry:139
m:   1,261,765 | c:  20,182 | String  poparser.ry:219
m:   1,011,760 | c:       4 | FastGettext::GetText::MOFile  /Users/joerafaniello/.gem/ruby/2.2.5/gems/fast_gettext-1.0.0/lib/fast_gettext/po_file.rb:14
you can see we have double the strings created by the poparser.ry
lazy loading unused message catalogs should drive that number down
Joe Rafaniello
@jrafanie
May 31 2016 14:13
another option is to try a different parser such as ruby-ll instead of racc parser, I have no idea how much work that is
(notice we have 4 MOFile objects created from our po_file, one for each message catalog)
Keenan Brock
@kbrock
May 31 2016 15:34
@jrafanie do we have a common method that takes an ugly class name and shortens it to something reasonable?
Jason Frey
@Fryguy
May 31 2016 15:34
example?
Joe Rafaniello
@jrafanie
May 31 2016 15:37
Yes, what @Fryguy said
Jason Frey
@Fryguy
May 31 2016 15:38
there is one method to get to a config symbol, but I'm not sure that's what you're looking for (or is the right tool for the job)
there is also the method to get you the ems_type, but that's specific to EMSes and also has a specific purpose
Keenan Brock
@kbrock
May 31 2016 15:38
Vmware::InfraManager::RefreshWorker ==> VMware::Refresh
just running out of characters on the screen
Jason Frey
@Fryguy
May 31 2016 15:39
no, there is nothing like that...
Keenan Brock
@kbrock
May 31 2016 15:39
thanks
Jason Frey
@Fryguy
May 31 2016 15:39
I guess we could create aliases, but then you run the risk of breaking things (e.g. what will you do when you have both a Vmware::InfraManager::RefreshWorker and a Vmware::CloudManager::RefreshWorker)
Keenan Brock
@kbrock
May 31 2016 15:39
right, for display purposes only
and that routine would know that openstack has both, so it would need to add that information just for openstack
Jason Frey
@Fryguy
May 31 2016 15:40
well for that we have description
Keenan Brock
@kbrock
May 31 2016 15:40
and not for everyone
graphite is displaying a number of queues
would like for the names to be nicer on the eyes
Jason Frey
@Fryguy
May 31 2016 15:40
ManageIQ::Providers::Vmware::InfraManager.description
=> "VMware vCenter"
however, I don't see that at the class level for RefreshWorker, but it might be at the instance level
there is also the worker's proctitle
Joe Rafaniello
@jrafanie
May 31 2016 15:45
where are you seeing this @kbrock ? as @Fryguy said there's the proctitle we can change easily
Keenan Brock
@kbrock
May 31 2016 15:54
@jrafanie we are shipping this to statsd from our app
Joe Rafaniello
@jrafanie
May 31 2016 15:54
I don't know what that means
Keenan Brock
@kbrock
May 31 2016 15:54
@jrafanie but maybe we would want to create a common "proctitle" method. so we could use that in many places
Joe Rafaniello
@jrafanie
May 31 2016 15:57
git grep setproctitle... there's only two places we set it
suggestions welcome
Keenan Brock
@kbrock
May 31 2016 15:57
thanks so much
Jason Frey
@Fryguy
May 31 2016 15:58
@kbrock Note that we don't control the proctitle for puma workers
Keenan Brock
@kbrock
May 31 2016 15:58
ooh - yea
we do a good job with what we have
Keenan Brock
@kbrock
May 31 2016 15:59
I just want to do something similar when we're shipping it off to collect metrics and stuff
I mean, I could complain. but you can always complain
I would like to mimic our current behavior in graphite/grafana/statsd
there is just so much data going through there. so the extra long names are tricky to grok
Joe Rafaniello
@jrafanie
May 31 2016 16:04
it's not set in stone, suggestions, PRs welcome with before/after pros/cons
Keenan Brock
@kbrock
May 31 2016 16:11
@jrafanie but again. I'm not trying to change the current naming. I'm trying to use the same algorithm elsewhere
but yes, PRs are good. Think I'll create one this week...
and, thanks
Joe Rafaniello
@jrafanie
May 31 2016 16:14
ok, yeah, I don't understand the context so an example demonstrating the pain might help make it clear
Alex Krzos
@akrzos
May 31 2016 17:49
Any reason why saving roles on an appliance might not spawn the workers it should?
I have two appliances
one that I had to restart evmserverd and when i changed his roles
he started the smart proxy workers as expected
the other appliance did not
Joe Rafaniello
@jrafanie
May 31 2016 17:50
There ees
Ugh
Alex Krzos
@akrzos
May 31 2016 17:51
it's a 5.6.0.8 appliance
Joe Rafaniello
@jrafanie
May 31 2016 17:51
There was a caching bug
ManageIQ/manageiq#8798
Alex Krzos
@akrzos
May 31 2016 17:52
this line finds the changes on the appliance that correctly added his new roles grep "MiqServer#log_role_changes" evm.log and on the offending appliance I get nothing when I grep for role changes
I thought 5.6.0.8 had that fix
Jason Frey
@Fryguy
May 31 2016 20:36
I noticed when we have all the workers running, my log is flooded with SELECT pg_backend_pid()
I'm not sure why though
Keenan Brock
@kbrock
May 31 2016 20:37
ugh - I remember when that was introduced and I looked into it a little, but too many :sparkles:
Jason Frey
@Fryguy
May 31 2016 20:38
yeah...it's like 5 or 6 times a second for me
I assume it's because we take a connection and put it back in the pool
cc @dmetzger57
Keenan Brock
@kbrock
May 31 2016 20:44
@Fryguy think the pid is part of the logging? (using memory that has not been refreshed recently)
Jason Frey
@Fryguy
May 31 2016 20:46
oh hmmm
Dennis Metzger
@dmetzger57
May 31 2016 21:33
I'll take a look