These are chat archives for ManageIQ/manageiq/performance

13th
May 2016
Joe Rafaniello
@jrafanie
May 13 2016 13:20
Hey @akrzos, RE: increased memory usage with darga / 5.6... is that across each worker or specific workers?
Dennis Metzger
@dmetzger57
May 13 2016 13:41
The following is from Alex’s data (think Alex is on PTO today) https://docs.google.com/spreadsheets/d/1CWYKPJBMQHsTNnk8n1wM9yQvDy7v6sJ57mhc_zKdzEI/edit#gid=0
Screen Shot 2016-05-13 at 9.39.17 AM.png
thats from the “idle” test
Joe Rafaniello
@jrafanie
May 13 2016 13:49
is that 5.5.2 (not 5.2.5)
Keenan Brock
@kbrock
May 13 2016 13:50
pretty sure that is 5.5.2
Dennis Metzger
@dmetzger57
May 13 2016 13:50
This message was deleted
5.6.0-5 compared to 5.5.2.5-3 based on the spreadsheet, missed the leading 5.
Joe Rafaniello
@jrafanie
May 13 2016 13:53
yeah, 5.2.5 is ruby 1.9.3 rails 3.2.13, I doubt it would be useful to compare to
Dennis Metzger
@dmetzger57
May 13 2016 14:11
Generic Worker.png
Dennis Metzger
@dmetzger57
May 13 2016 14:27
Not enough coffee, the numbers I showed above are RSS not PSS. If anyone is interested I can post the PSS which of course is the more relevant / useful data :worried: , perhaps I need new glasses (P => R)
Jason Frey
@Fryguy
May 13 2016 14:30
PSS please :)
Dennis Metzger
@dmetzger57
May 13 2016 15:47
Here’s the PSS usage delta between 5.5.0.13-2 and 5.6.0-6 on an idle appliance (no provider added). Possibly next will be to get the same data set for after a provider has been added. Good news is a couple workers shrank, the bad news is … well the overall growth
PSS Delta.png
Joe Rafaniello
@jrafanie
May 13 2016 16:02
@dmetzger57 can you tests with upstream branches?
the ManageIQ/manageiq#8525 drops Rails console memory usage by ~12 MB
that should be seen in all workers
Even in production, we probably won't ever have all 4 message catalogs in use at one time
and even if we do, it will probably only be in UI workers
Oleg Barenboim
@chessbyte
May 13 2016 16:04
in the future, we could have dozens of translations
so, this fix will be even more important
Joe Rafaniello
@jrafanie
May 13 2016 16:05
Note, this is the direction of fastgettext
they'll be doing lazy load by default
the only problem that @himdel ran into was we store the friendly name of the message catalog within the catalog, so I had to hack it to not give a friendly name in the UI
he said switching catalogs had less than 1 second delay
Note, if ruby was really copy on write friendly, this wouldn't have too much improvement (we'd save only the cost of loading the not used catalogs in the single server process)
but it's heap allocation strategy is not CoW friendly so I bet over the span of an hour, most if not all of the objects from the catalogs have been copied to private OS pages in every worker
So, anyway, I wonder if this WIP PR made any dent in the idle workers increased memory usage
Joe Rafaniello
@jrafanie
May 13 2016 16:13
I don't know how doable lazy loading the routes.rb is... but that's accounting for 10 MB of memory and 1 second in rails boot process... I"d imagine that is also in all worker processes
Also, @dmetzger57, I don't know that our GC parameters are optimized for appliance usage to minimize memory usage while also not greatly increasing GC time. It's been a while and those number might not be accurate anymore.
ok, going to lunch, then will be on and offline as I work at my daughter's appointment
Jason Frey
@Fryguy
May 13 2016 16:19
feels like something that should be easy to lazy load
Chris Arcand
@chrisarcand
May 13 2016 16:20
IIRC lazy loading routes in Rails has been a long requested feature that’s never been able to be done for reasons™, isn’t it?
Jason Frey
@Fryguy
May 13 2016 16:23
reasons™ on our side? or reasons™ on Rails' side?
I know of nothing on our side, which is why I ask
Joe Rafaniello
@jrafanie
May 13 2016 19:24
So, I looked into why lazy loaded routes haven't happened
1) rails setups a routes_reloader in initializers to reload routes in dev mode if routes.rb changes in the same way that classes get reloaded
2) other rails initializers call app.routes.append or other things to add such things as the rails welcome page
these things expect the routes to be eager loaded
Joe Rafaniello
@jrafanie
May 13 2016 19:29
it's gnarly
it might be possible to dump your serialized route_sets to avoid loading it as part of boot