These are chat archives for ManageIQ/manageiq/performance

16th
Oct 2015
Matthew Draper
@matthewd
Oct 16 2015 14:59
I guess we need some new numbers, to see where we're at. @dmetzger57 will you be able to re-run your thing, with some of the changes we've got floating around?
Oleg Barenboim
@chessbyte
Oct 16 2015 15:00
are any of those Performance PRs ready to be merged?
Jason Frey
@Fryguy
Oct 16 2015 15:00
which ones?
Dennis Metzger
@dmetzger57
Oct 16 2015 15:00
@matthewd yep, just need a branch to run
Oleg Barenboim
@chessbyte
Oct 16 2015 15:01
#4911, #4912, and #4894
Jason Frey
@Fryguy
Oct 16 2015 15:02
#4912 I want the comment back, but code-wise I think it's good...@matthewd wanted to see real numbers first before doing any other changes
Matthew Draper
@matthewd
Oct 16 2015 15:02
ManageIQ/manageiq#4911 should be fine
The only failure is known, elsewhere, and since fixed
Jason Frey
@Fryguy
Oct 16 2015 15:03
#4911 is fine even though it failed for unrelated bug
Matthew Draper
@matthewd
Oct 16 2015 15:03
I've been cleaning up #4912 today… should have a shinier version shortly
Jason Frey
@Fryguy
Oct 16 2015 15:03
#4894 has the same known unrelated failure, so it's good to merge
Joe Rafaniello
@jrafanie
Oct 16 2015 15:19
@dmetzger57 want to run with the GC tuning?
it’s clearly not the optimal numbers but the out of box 2.2.3 kills you if you have any large loops of high allocations
Oleg Barenboim
@chessbyte
Oct 16 2015 15:20
@matthewd ManageIQ/manageiq#4912 needs a rebase
Dennis Metzger
@dmetzger57
Oct 16 2015 15:22
@jrafanie i'll ru without turning the knobs first, then let me know what knobs to turn and i'll run that way - we can then see the impact
Joe Rafaniello
@jrafanie
Oct 16 2015 15:22
ok
@dmetzger57 I’ve been using this at the bottom of /etc/default/evm and restarting or exiting my ssh shell:
export RUBY_GC_HEAP_GROWTH_MAX_SLOTS=300000
export RUBY_GC_HEAP_INIT_SLOTS=600000
export RUBY_GC_HEAP_GROWTH_FACTOR=1.25
be back shortly
Dennis Metzger
@dmetzger57
Oct 16 2015 15:26
will try
Jason Frey
@Fryguy
Oct 16 2015 16:05
awesome description
I love that we no longer have to patch const_missing
Always hated that
Matthew Draper
@matthewd
Oct 16 2015 17:16
@dmetzger57 once you've got your baseline run, and one with @jrafanie's tuning, can you please do one with ManageIQ/manageiq#4912 ?
Dennis Metzger
@dmetzger57
Oct 16 2015 17:19
@matthewd i'm running 4911, 4912 and 4894 now through small/medium/large environments
Matthew Draper
@matthewd
Oct 16 2015 17:19
Ah, excellent
Dennis Metzger
@dmetzger57
Oct 16 2015 17:36
ok, i'm gonna test these 1 at a time. the large environment is at 2,450 seconds elapsed and still doing the inventory, with out the patches this took 280 seconds. hopped on and the appliance is swapping like crazy.
Matthew Draper
@matthewd
Oct 16 2015 17:37
I think just the smalls should give some useful insight
Dennis Metzger
@dmetzger57
Oct 16 2015 17:41
i want to verify there's nothing crazy happening in my environment before I trust the numbers from these runs, given that the large run just went out to lunch - I've not had a run in the large environment take more than 298 seconds before this so I'm a little surprised.
Joe Rafaniello
@jrafanie
Oct 16 2015 17:50
@matthewd is there a 4-2-stable release coming or are you secretly hoping we find more fixes to backport to rails?
I don’t know if any of @tenderlove’s changes on master for reduced allocations are needed on 4.2 stable
Matthew Draper
@matthewd
Oct 16 2015 17:51
I think there was one — it was just the rest of the things he found, that ended up only affecting master
Jason Frey
@Fryguy
Oct 16 2015 17:51
According to his numbers, they are not needed
Matthew Draper
@matthewd
Oct 16 2015 17:54
As for a release, I was just thinking about that earlier. We should be able to get one in in time… though it may only be rc when we hit rc upstream
Joe Rafaniello
@jrafanie
Oct 16 2015 17:58
does it make sense to track stable for now in the Gemfile? I don’t think the leak you fixes in 4.2-stable is huge but it feels weird to have things fixed but not test with them
I’ve been tracking 4-2-stable in my test appliance I’m generating numbers against
Matthew Draper
@matthewd
Oct 16 2015 18:00
I guess that's something we should compare — whether it actually gives a gain worthy of the inelegance
Jason Frey
@Fryguy
Oct 16 2015 18:01
@jrafanie Out of curiosity, is Rails console cheaper with @matthewd's deferral change?
Joe Rafaniello
@jrafanie
Oct 16 2015 18:02
it depends if it hits the require ;-)
Matthew Draper
@matthewd
Oct 16 2015 18:02
It certainly should be; that's what I was measuring with
Well, I was using runner… and forcing a load of a model
Joe Rafaniello
@jrafanie
Oct 16 2015 18:15
@Fryguy is there some magic to a rhev ems to start collecting inventory via console/runner? Your trick to do vmware doesn’t work if I add the rhev ems and give it validated creds
Jason Frey
@Fryguy
Oct 16 2015 18:16
that trick is very specific to vmware because of the broker stuff
for a rhev ems, just EmsRefresh.refresh(e) is all you need
Joe Rafaniello
@jrafanie
Oct 16 2015 18:19
Sorry, I got confused… :confused:
what I meant is I add the rhev and vmware ems’s with verified creds, then start the server and the rhev refresh isn’t running
vmware runs, not rhev
Oleg Barenboim
@chessbyte
Oct 16 2015 18:21
are both assigned to the same zone?
Joe Rafaniello
@jrafanie
Oct 16 2015 18:21
yes
I guess I can fire up the UI and add the rhev provider but I’d rather do it scripted
Matthew Draper
@matthewd
Oct 16 2015 18:22
I don't know anything useful about workers… but could your single-workers-whatever initializer be interfering?
Jason Frey
@Fryguy
Oct 16 2015 18:22
that doesn't make sense
Joe Rafaniello
@jrafanie
Oct 16 2015 18:22
@matthewd good point.. thanks, I removed that from my testing anyway
Jason Frey
@Fryguy
Oct 16 2015 18:22
oh yeah...check your hack (or use MIQ_SPARTAN instead)
Joe Rafaniello
@jrafanie
Oct 16 2015 18:24
I’ll take a look, I want to look at alex’s findings: RHEVM medium provider - Between 184MiB to 195MiB on 5.4 going to RHEVM medium provider - Between 531MiB to 577MiB on 5.5
Jason Frey
@Fryguy
Oct 16 2015 18:27
o_O
Joe Rafaniello
@jrafanie
Oct 16 2015 18:39
@matthewd matthewd-very-lazy-loading nice
Oleg Barenboim
@chessbyte
Oct 16 2015 18:39
@jrafanie does it work?
Joe Rafaniello
@jrafanie
Oct 16 2015 18:39
checking
Matthew Draper
@matthewd
Oct 16 2015 18:40
@jrafanie it's more like "actually lazy loading instead of just pretending", but that was too long ;)
Joe Rafaniello
@jrafanie
Oct 16 2015 18:40
well, at least I’m checking if ruby parser is loaded in console and how the affects objects
@matthewd is there a limit on branch names?
Matthew Draper
@matthewd
Oct 16 2015 18:40
There is when I have to type them :grin:
Joe Rafaniello
@jrafanie
Oct 16 2015 18:44
@matthewd 20,000 less live objects with your PR, no ruby_parser in $LOADED_FEATURES… so as long as you load it at the right time and we have the result of ruby_parser cached somewhere so we don’t need to parse our files, it’s a win ;-)
Jason Frey
@Fryguy
Oct 16 2015 18:45
merge it!
Joe Rafaniello
@jrafanie
Oct 16 2015 18:46
FYI, @Fryguy I’m wondering if we should keep the GC init_slots at 600_000 if console has ~460_000 live objects
but we can tweak that if we need to
GC.start; GC.stat.values_at(:total_allocated_objects, :total_freed_objects).reduce(:-)
=> 464769
Matthew Draper
@matthewd
Oct 16 2015 18:47
@jrafanie what about memory? I was seeing a 20 MB difference.
Oh, but that's after you reference a model — so the old one would've done the full cascade load
Joe Rafaniello
@jrafanie
Oct 16 2015 18:49
yeah, what woudl be the best test? idle workers?
keep in mind if you’re using 2.2.3 untuned it’s most likely much small than 20 MB difference since it loves to eat your memory
Matthew Draper
@matthewd
Oct 16 2015 18:50
For reference, I was doing: RAILS_ENV=production be rails r 'DescendantLoader.status; ExtManagementSystem.subclasses; DescendantLoader.status; VmInfra.subclasses; DescendantLoader.status; Vm.subclasses; DescendantLoader.status'
You'll obviously want something a bit less artificial ;)
Jason Frey
@Fryguy
Oct 16 2015 18:54
@jrafanie What number were you thinking instead?
higher or lower?
Joe Rafaniello
@jrafanie
Oct 16 2015 18:57
you know what… I have a bunch of .csvs with all those values… so i can graph total_live_objects
Joe Rafaniello
@jrafanie
Oct 16 2015 19:13
maybe 600_000 is good @Fryguy, idle automate workers range from about 600_000 to 700_000
(live or what the GC things are live objects)
even schedule worker starts at about that 600k object count magic number
Oleg Barenboim
@chessbyte
Oct 16 2015 19:14
wow - 600_000 objects just to get going?
Joe Rafaniello
@jrafanie
Oct 16 2015 19:14
goes up to 900k and hovers in the 650k-900k range
yup
that’s nothing
the vmware refresh worker gets to almost 6 million “live” objects before Jason’s PR
that’s why 2.2.3 is so different for us… that worker gets in the 5-6 million live objects range, nearly 3-4 million being “old” objects… they don’t get collected until a full GC happens
Matthew Draper
@matthewd
Oct 16 2015 19:18
.. because we gather every single thing VMware knows about every single thing, and store it all in a big hash :neutral_face:
Oleg Barenboim
@chessbyte
Oct 16 2015 19:19
there has to be a better way
are we seeing similar issues on RHEV or OpenStack?
Matthew Draper
@matthewd
Oct 16 2015 19:19
I suspect the short better way is to only keep the information we end up using
Joe Rafaniello
@jrafanie
Oct 16 2015 19:20
yeah, trying to get that working… Alex saw it with rhev also
Matthew Draper
@matthewd
Oct 16 2015 19:20
I haven't actually confirmed, but surely we don't use all the info vmware gives us
Joe Rafaniello
@jrafanie
Oct 16 2015 19:20
need to also try cap and u
also, taking Jason’s PR, releasing the vc_data as soon as possible
Matthew Draper
@matthewd
Oct 16 2015 19:21
Right now we build a hash represenation of vmware's soap/xml response… in the other providers, I think we build a hash that's closer to our target final form
Oleg Barenboim
@chessbyte
Oct 16 2015 19:21
@jrafanie - I thought Jason's PR is merged already
Joe Rafaniello
@jrafanie
Oct 16 2015 19:21
yes, it was merged
need to do this area too… https://github.com/ManageIQ/manageiq/blob/da90f75d9d5a7e9bcbd8cc6e4a4018269a5e4105/gems/pending/VMwareWebService/MiqVimUpdate.rb#L35-L45, line 39 is run for each CI in our system, 3000 vms, 100 hosts, 30 storages, etc… all the while we’re copying the raw data into the broker cache, while keeping the original objects holding the reference to the raw data so it can’t be collected until we finish all of the objects
that’s for alex’s large vmware environment
Jason Frey
@Fryguy
Oct 16 2015 19:30

there has to be a better way

Yeah, skeletal refresh

I suspect the short better way is to only keep the information we end up using

We already do that...the broker has a filter, then there is a selector spec which is another kind of filter

Matthew Draper
@matthewd
Oct 16 2015 19:31
Isn't that measurably after we've hashified the whole lot, though? :confused:
Jason Frey
@Fryguy
Oct 16 2015 19:31
the hash in the middle has only what we need...I've tried to be very selective about that
Matthew Draper
@matthewd
Oct 16 2015 19:33
I guess I'm wondering why we don't do that step, straight from the XML, without the first round of hashes+strings+VimStrings
Jason Frey
@Fryguy
Oct 16 2015 19:34
the broker can't be that selective when asking VC for updates, otherwise VC's own performance is hurt...so we use this prop map: https://github.com/ManageIQ/manageiq/blob/master/gems/pending/VMwareWebService/VimPropMaps.rb#L146
if you compare, they are pretty close
we use the broker for more than refresh, so it needs to have data that covers all cases
I think the broker has clear memory issues...we've thought that forever... @roliveri insists the actual inventory is relatively small (10s to maybe 100 megs), but the broker memory is always in 1+ GB range
I definitly think it warrants some deeper introspection
Dennis Metzger
@dmetzger57
Oct 16 2015 21:14
I still don't know what's up with the Patched Master in the Large environment, but the Small environment numbers are solid and charted here https://docs.google.com/a/redhat.com/spreadsheets/d/1AuWfmQztEP94j7J1CIi3obl5tc5lz5nQjIS4RYgoPk0/edit?usp=sharing (I'll chart the Medium environment later) As you can see the 3 PRs do make a difference in the small environment - the ~1Gb larger base footprint is a separate issue.
Matthew Draper
@matthewd
Oct 16 2015 21:21
@dmetzger57 does this include @jrafanie's GC config tweaks? On the patched version, or both?
Joe Rafaniello
@jrafanie
Oct 16 2015 21:23
Current status: Garbage collection took 3.3411167800002204 seconds
Dennis Metzger
@dmetzger57
Oct 16 2015 21:24
the patched version contains 4894, 4911 and 4912; it does not include the GC tuning - thats to come
Joe Rafaniello
@jrafanie
Oct 16 2015 21:24
rhevm large causes the refresh worker to hit 1.131g resident
about to dig into the rhev refresh side now that I have debug logging and object data
Matthew Draper
@matthewd
Oct 16 2015 21:25
Oh, so we might've actually earned that gap in the footprint?
300 MB down, 800 MB to go ¯\_(ツ)_/¯
I guess it depends what the GC change does… I think that'll buy us 300-400 off the old figure… but those gains likely aren't additive
Joe Rafaniello
@jrafanie
Oct 16 2015 21:34
Ok, so I have some large rhev refresh worker graphs...
master, no tuning...
manageiq_providers_redhat_infra_manager_refresh_worker_3434-memory_usage.png
guess what’s the biggest object type?
Matthew Draper
@matthewd
Oct 16 2015 21:37
Is that master from before or after Jason's PR was merged?
Joe Rafaniello
@jrafanie
Oct 16 2015 21:37
after, his was vmware, right?
Jason Frey
@Fryguy
Oct 16 2015 21:37
my PR should not affect rhev
Joe Rafaniello
@jrafanie
Oct 16 2015 21:37
this is redhat
Matthew Draper
@matthewd
Oct 16 2015 21:37
Oh, right
Joe Rafaniello
@jrafanie
Oct 16 2015 21:37
manageiq_providers_redhat_infra_manager_refresh_worker_3434-T_OBJECT_SIZE.png
manageiq_providers_redhat_infra_manager_refresh_worker_3434-heap_live_slots.png
Jason Frey
@Fryguy
Oct 16 2015 21:38
you're killing me not stacking that last one ;)
haha
Joe Rafaniello
@jrafanie
Oct 16 2015 21:38
haha, I knew you’d like that one
here’s another just for you
manageiq_providers_redhat_infra_manager_refresh_worker_3434-live_objects.png
Matthew Draper
@matthewd
Oct 16 2015 21:38
How large is large again?
Joe Rafaniello
@jrafanie
Oct 16 2015 21:39
3000 vm, 100 hosts, 121 storages
Jason Frey
@Fryguy
Oct 16 2015 21:39
OMG 3000 on RHEV
Matthew Draper
@matthewd
Oct 16 2015 21:39
We're producing & retaining 200M hashes. :neutral_face:
Jason Frey
@Fryguy
Oct 16 2015 21:39
hahahahaha
that's like a zillion API calls
Joe Rafaniello
@jrafanie
Oct 16 2015 21:40
@matthewd that’s hash size, not count
Jason Frey
@Fryguy
Oct 16 2015 21:40
even if we just queried it all and threw it away it would be enormous
Matthew Draper
@matthewd
Oct 16 2015 21:40
ohhh, okay
Jason Frey
@Fryguy
Oct 16 2015 21:40
there might be some GC wins similarly to VMware there though
Matthew Draper
@matthewd
Oct 16 2015 21:40
Maybe I should read the titles :P
Joe Rafaniello
@jrafanie
Oct 16 2015 21:41
Note, the Ovirt::Inventory.refresh gets it to the initial 600-700 MB resident
Jason Frey
@Fryguy
Oct 16 2015 21:42
yeha that's just collecting the data
Joe Rafaniello
@jrafanie
Oct 16 2015 21:42
the rest has some good opportunities I think
Matthew Draper
@matthewd
Oct 16 2015 21:42
Yeah, we really need to find a way to not need all the data in memory at once
Jason Frey
@Fryguy
Oct 16 2015 21:43
SO the RHEV one might be able to be changed into a fetch/parse style parser
like amazon
where we query the data and parse it as we go along
what makes that hard is that because the RHEV API is terrible we had to collect the data in parallel, and putting that inside the parser might be tough
Matthew Draper
@matthewd
Oct 16 2015 21:44
I'd imagine vmware could too… even if it fetches the whole blob of XML at once, we could pipeline the processing
Joe Rafaniello
@jrafanie
Oct 16 2015 21:44
7.5 million live objects of which nearly 1/2 are young
even minor GCs would be expensive
Matthew Draper
@matthewd
Oct 16 2015 21:45
As mentioned before.. the XML string itself isn't huge.. it's only when we turn that into eleventy billion separate objects that things get unpleasant
Jason Frey
@Fryguy
Oct 16 2015 21:45
maybe, but the data is queried differently on vmware...we get all the vms at once from the broker.
Oleg Barenboim
@chessbyte
Oct 16 2015 21:45
@matthewd "eleventy" - what kind of number is that?
Joe Rafaniello
@jrafanie
Oct 16 2015 21:46
can you check my math on GC.stat though…
Matthew Draper
@matthewd
Oct 16 2015 21:46
@chessbyte "obviously fictitous" :)
Joe Rafaniello
@jrafanie
Oct 16 2015 21:46
    live_objects  = gc_stat[:total_allocated_objects] - gc_stat[:total_freed_objects]
    young_objects = live_objects - gc_stat[:old_objects]
Matthew Draper
@matthewd
Oct 16 2015 21:46
but… like… lots.
Oleg Barenboim
@chessbyte
Oct 16 2015 21:47
@Fryguy thanks for the reference
Gregg Tanzillo
@gtanzillo
Oct 16 2015 21:50
Bilbo Baggins was 111 years old on his elventy-frist birthday :)
Joe Rafaniello
@jrafanie
Oct 16 2015 21:51
:ring:
Keenan Brock
@kbrock
Oct 16 2015 21:56
I just keep thinking of sax vs dom. (or the newer xml pull parser)
Joe Rafaniello
@jrafanie
Oct 16 2015 21:56
@matthewd was is just 4912 that wasn’t merged that you wanted me to try?
Keenan Brock
@kbrock
Oct 16 2015 21:57
and rails active record find in batches
Matthew Draper
@matthewd
Oct 16 2015 21:57
@jrafanie maybe?
Joe Rafaniello
@jrafanie
Oct 16 2015 21:57
I tested with master containing 4894(vmware refresh for gc), and 4911 (inverse_of)
Matthew Draper
@matthewd
Oct 16 2015 21:57
I mean, that is the only one, but I'm not sure how much it'll help you
It should give a couple of meg reduction to the baseline memory consumption, and that's it
Joe Rafaniello
@jrafanie
Oct 16 2015 21:58
ok, that’s not going to help 1.1 GB too much ;-)
Matthew Draper
@matthewd
Oct 16 2015 21:58
Yeah, it only really gets interesting when applied across 15 processes
Joe Rafaniello
@jrafanie
Oct 16 2015 21:59
yes, for sure
how about the routes? Is there a way to disable loading that in runner scripts?
it’s enormous ;-)
Matthew Draper
@matthewd
Oct 16 2015 22:00
Well, I guess we could hack it by just wrapping them with an if ui_worker conditional
Keenan Brock
@kbrock
Oct 16 2015 22:00
@jrafanie :scissors: we could cut down our routes... then everyone would benifit. (just sayin')
Joe Rafaniello
@jrafanie
Oct 16 2015 22:01
yes, we could do that @kbrock and should
Keenan Brock
@kbrock
Oct 16 2015 22:01
wc -l config/routes.rb 
    2064 config/routes.rb
sorry - unneeded snark
Matthew Draper
@matthewd
Oct 16 2015 22:02
restfulizing them should help a lot with the line count… though maybe not the memory
Jason Frey
@Fryguy
Oct 16 2015 22:02
why is routes so expensive anyway
Are route objects that big?
Joe Rafaniello
@jrafanie
Oct 16 2015 22:02
it was 7 MB loading routes.rb
Matthew Draper
@matthewd
Oct 16 2015 22:03
How expensive is it? :confused:
Joe Rafaniello
@jrafanie
Oct 16 2015 22:03
several days ago
/Users/joerafaniello/Code/manageiq/config/routes.rb (0.709244) RES:7.0234375 VIRT:7.0
Jason Frey
@Fryguy
Oct 16 2015 22:03
does it load anything underneath it?
Joe Rafaniello
@jrafanie
Oct 16 2015 22:03
3/4 second to load it on SSD
Matthew Draper
@matthewd
Oct 16 2015 22:04
I think it does some precompilationy-type stuff to ensure routing is fast at runtime… but nothing crazy, I don't think
Jason Frey
@Fryguy
Oct 16 2015 22:04
7MB / 2000 routes = ~3.5 KB per route?
that just feels like a lot more than it should
Joe Rafaniello
@jrafanie
Oct 16 2015 22:05
to be honest though, delay loading the message catalogs if possible is also in the same ballpark
I believe we had 6-8 MB of message catalogs of nearly 15 MB in fast_gettext initializer
/Users/joerafaniello/Code/manageiq/config/initializers/fast_gettext.rb (0.904699) RES:14.9140625 VIRT:13.8203125
Matthew Draper
@matthewd
Oct 16 2015 22:06
@Fryguy yeah, ~80 objects per route does seem high
Oh, unless we're defining route helpers etc for them?
Jason Frey
@Fryguy
Oct 16 2015 22:07
@jrafanie I can't remember but how much memory was just 1 message catalog (that is rm all the others and get the memory)
I don't think we have route helpers
Matthew Draper
@matthewd
Oct 16 2015 22:07
We may not be using them…
Jason Frey
@Fryguy
Oct 16 2015 22:08
fair enough
Matthew Draper
@matthewd
Oct 16 2015 22:08
I actually don't remember whether we automatically define them for "plain" routes, or only fancier ones
Joe Rafaniello
@jrafanie
Oct 16 2015 22:08
yeah, i’m not in position to test it now, but I believe it dropped 6-8 MB
Jason Frey
@Fryguy
Oct 16 2015 22:08
so base gettext is still 7 MB or so
we're still talking peanuts though
We need a treemap view of memory
gonna try to whip one of those up
Matthew Draper
@matthewd
Oct 16 2015 22:10
Our baseline growth is only ~73 MB per process
(1100 MB / 15 processes)
Jason Frey
@Fryguy
Oct 16 2015 22:11
but that's the delta
we're not taking into account gems that were removed
Matthew Draper
@matthewd
Oct 16 2015 22:11
True. But the pile of peanuts does start to add up.
Jason Frey
@Fryguy
Oct 16 2015 22:12
agreed
By peanuts I was more talking from the entire codebase...the baseline even on 5.4 seems ridiculously high to me
Matthew Draper
@matthewd
Oct 16 2015 22:14
To me it doesn't seem much far outside "large rails app"… it's the multiplier that kills us
Dennis Metzger
@dmetzger57
Oct 16 2015 22:15
VMware Small Env.png
Matthew Draper
@matthewd
Oct 16 2015 22:17
Well that's looking better
Dennis Metzger
@dmetzger57
Oct 16 2015 22:19
just the initial gap :worried:
Joe Rafaniello
@jrafanie
Oct 16 2015 22:20
So, this is still with extra workers, right? Automate workers which didn’t exist on 5.4, right?
Jason Frey
@Fryguy
Oct 16 2015 22:21
how much of that extra gap == automate? and how much doesn't
Matthew Draper
@matthewd
Oct 16 2015 22:22
I think the automate workers should be down to about 150MB each now… so, ~ half of the remainder?
Joe Rafaniello
@jrafanie
Oct 16 2015 22:23
my automate workers consumed ~ 170 RSS
Joe Rafaniello
@jrafanie
Oct 16 2015 22:36
another option… force full GC more frequently, we do GC.start/ObjectSpace.garbage_collect every 15 minutes on workers, never explicitly in the server
we’ll have to throw cap and u into the mix next week… @Fryguy let’s try to get that broker thing on monday: https://github.com/ManageIQ/manageiq/blob/da90f75d9d5a7e9bcbd8cc6e4a4018269a5e4105/gems/pending/VMwareWebService/MiqVimUpdate.rb#L35-L45
have a good weekend everyone
Ah, @dmetzger57 I just saw your chart… not sure how I missed it… so the GC tuning looks to get us ~250 MB back on a system
cool
Dennis Metzger
@dmetzger57
Oct 16 2015 22:41
yes indeed, another win
Joe Rafaniello
@jrafanie
Oct 16 2015 22:46
cool, more fun next week ;-)
Jason Frey
@Fryguy
Oct 16 2015 22:46
@jrafanie also schedule_worker