These are chat archives for ManageIQ/manageiq/performance

22nd
Oct 2015
Dennis Metzger
@dmetzger57
Oct 22 2015 00:10
Rss_Footprint.pdf
CFME_Worker_Sizes.pdf
Master_Worker_Sizes.pdf
Dennis Metzger
@dmetzger57
Oct 22 2015 12:27
vSphere Small Env Worker Growth.pdf
Alex Krzos
@akrzos
Oct 22 2015 13:02
@dmetzger57 Can you point me to the GC tuning for master in that setup?
Dennis Metzger
@dmetzger57
Oct 22 2015 13:04
@akrzos here's what I have in /etc/default/evm based on input from @jrafanie
Jason Frey
@Fryguy
Oct 22 2015 13:04
It's part of the appliance now so after pull and reboot you should have it. If not see https://github.com/ManageIQ/manageiq-appliance/pull/36/files
Dennis Metzger
@dmetzger57
Oct 22 2015 13:04
export RUBY_GC_HEAP_GROWTH_MAX_SLOTS=300000
export RUBY_GC_HEAP_INIT_SLOTS=600000
export RUBY_GC_HEAP_GROWTH_FACTOR=1.25
or just what @Fryguy said
Jason Frey
@Fryguy
Oct 22 2015 13:05
:). I figured the explanation in the link might be nice to know
Alex Krzos
@akrzos
Oct 22 2015 13:06
@dmetzger57 and @Fryguy Yeah I was wondering if my updated master appliance had it or not, and didn't see the exports in /etc/default/evm, nice to know it's in evm_ruby
Joe Rafaniello
@jrafanie
Oct 22 2015 14:11
Yes, sorry, @akrzos, I forgot to mention that it went into a different file on the appliance
the shortcut of putting it in the /etc/default/evm would also work
Joe Rafaniello
@jrafanie
Oct 22 2015 14:25
@akrzos Is the large vsphere environment up and running?
Alex Krzos
@akrzos
Oct 22 2015 14:25
@jrafanie Should be, let me check that env
@jrafanie It's still up
Joe Rafaniello
@jrafanie
Oct 22 2015 14:26
thanks, maybe my setup is screwy
Alex Krzos
@akrzos
Oct 22 2015 14:26
if you want the worse case scenario, you can always go with xlarge (10k vms) :smirk:
Keenan Brock
@kbrock
Oct 22 2015 14:44
poor little providers. they don't stand a chance now that @akrzos is back
Dennis Metzger
@dmetzger57
Oct 22 2015 14:48
@akrzos so you don't like @jrafanie - sending him to the xlarge .....
Alex Krzos
@akrzos
Oct 22 2015 14:49
@dmetzger57 :smile: Imagine the scale if we can make the default appliance memory work with xlarge!
Keenan Brock
@kbrock
Oct 22 2015 14:50
coach @akrzos says, "don't just win the game people. destroy the enemy."
Joe Rafaniello
@jrafanie
Oct 22 2015 14:52
what's the ip of xlarge?
.68?
Alex Krzos
@akrzos
Oct 22 2015 14:53
Yes
Joe Rafaniello
@jrafanie
Oct 22 2015 14:53
I'm able to run the broker locally and do a refresh manually in a rails runner script (outside of the refresh worker) and it seems to duplicate the memory growth
Alex Krzos
@akrzos
Oct 22 2015 14:54
@jrafanie Without @dmetzger57 bulk patch your looking at >90m on initial refresh with that env
At least that was with 5.4
Joe Rafaniello
@jrafanie
Oct 22 2015 14:54
nice
Keep in mind, we're still not running with rails 4-2-stable on master
@Fryguy tried to fix the hakiri failure by rerunning it to no avail: ManageIQ/manageiq#5025
Jason Frey
@Fryguy
Oct 22 2015 15:00
I wonder if the AR bump causes Hakiri to break
Matthew Draper
@matthewd
Oct 22 2015 15:05
@akrzos I'm still struggling to imagine us making the default appliance memory work with large :trollface:
Dennis Metzger
@dmetzger57
Oct 22 2015 16:11
@akrzos @jrafanie @matthewd my last run against the large vmware environment (using a 6Gb appliance with a git pull this morning) took under 15 minutes.
that was under 15 minutes for the initial provider refresh
Joe Rafaniello
@jrafanie
Oct 22 2015 16:19
@matthewd care to look at this at your leisure? https://gist.github.com/jrafanie/fb7873440f1117d41078 , code here: https://github.com/rails/rails/compare/4-2-stable...jrafanie:naively_retain_separators, note the router in master doesn't have this problem so it would require either backporting master changes of the router or a hack similar to mine
Alex Krzos
@akrzos
Oct 22 2015 16:20
@dmetzger57 on untuned, 5.5.0.3 i have initial refresh taking 382s 99th percentile of 4 iterations on vmware large
Matthew Draper
@matthewd
Oct 22 2015 16:22

@jrafanie can we monkey-patch it with something like

def (ActionDispatch::Routing::SEPARATORS.freeze).join
  @joined ||= super
end

?

(obviously uglier, and much greater chance of blowing up in your face… but possibly surgical enough that we could carry it without duplicating the implementation of initialize, say)
Dennis Metzger
@dmetzger57
Oct 22 2015 16:44
@akrzos what was the memory size on your 5.5.0.3 run?
Alex Krzos
@akrzos
Oct 22 2015 16:46
@dmetzger57 1152MiB, keep in mind this is RSS change (End of Run - Console start up cost) (99th percentile measurement)
so with the console which should be around another 170MiB
Dennis Metzger
@dmetzger57
Oct 22 2015 16:49
@akrzos my 8Gb Master appliance runs took an average of 295 seconds. I'm going to grab a fresh appliance and retry with 6Gb.
Alex Krzos
@akrzos
Oct 22 2015 16:50
@dmetzger57 I imagine we have different cpus between where our appliances are running
My testbed is on Sandybridge Intel cpus
I have haswell too
but haven't run the benchmarks on them yet
Dennis Metzger
@dmetzger57
Oct 22 2015 16:53
Mine are run from my Macbook via Fusion, so the CPU is an Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Alex Krzos
@akrzos
Oct 22 2015 16:54
That's haswell
not sure of what overhead fusions would add, compared to KVM with RHEV that I have
Joe Rafaniello
@jrafanie
Oct 22 2015 17:06
@matthewd I have to digest your change , but yes , a monkey patch could be worth it
Matthew Draper
@matthewd
Oct 22 2015 17:19
@jrafanie 2200 tiny strings should not be worth 370k, though :confused:
Jason Frey
@Fryguy
Oct 22 2015 17:26
@matthewd , no but we figured, hey why not
We looked into routes memory, btw, aside from those strings
Each route creates a regex
We could consolidate a few, but not enough to really matter
Maybe knocks routes regex memory by under a MB
Would be nice to find a way for a runner process to load routes only when needed
Matthew Draper
@matthewd
Oct 22 2015 17:29
In principle, runners shouldn't need to load AC/AV at all
(unless they send mail, I guess?)
Jason Frey
@Fryguy
Oct 22 2015 17:31
Yeah, only generic workers might send mail
With notifier role
That might be a big win
Joe Rafaniello
@jrafanie
Oct 22 2015 17:44
yeah, @matthewd, i don't understand why the memsizeofall(String) reports ~400 kb from a small number of strings ¯\(ツ)
Jason Frey
@Fryguy
Oct 22 2015 17:50
@matthewd that :point_up: October 22, 2015 12:22 PM is some of those kookiest code I've ever seen :D
Joe Rafaniello
@jrafanie
Oct 22 2015 17:51
Did someone say :cookie: ?
Matthew Draper
@matthewd
Oct 22 2015 18:01
@Fryguy thanks, I think :)
A quick hack of an attempt to run the refresher without AC/AV gives about 10mb savings
But it's messy, and that's not going to apply to all the processes… ¯\_(ツ)_/¯
Joe Rafaniello
@jrafanie
Oct 22 2015 18:05
well, it would be for most processes… in that 10MB run, did you also remove hamlit, sass and other AC/AV related gems not needed by most workers?
Matthew Draper
@matthewd
Oct 22 2015 18:05
Yes
Had to, because deps
Joe Rafaniello
@jrafanie
Oct 22 2015 18:06
ok
i’m surprised it’s only 10
¯_(ツ)_/¯
Matthew Draper
@matthewd
Oct 22 2015 18:07
I think there'd be some more you could gain if you dropped the fundamental rails-app-ness, and ran as a pure happens-to-be-using-AR-models ruby script
Joe Rafaniello
@jrafanie
Oct 22 2015 18:07
yeah
Matthew Draper
@matthewd
Oct 22 2015 18:07
And actually, I was chasing AC… it's possible AD was still loading, maybe
Joe Rafaniello
@jrafanie
Oct 22 2015 18:08
doesn’t seem worth the time if we plan to take advantage of COW with fork
Matthew Draper
@matthewd
Oct 22 2015 18:08
My thinking exactly
Rearranging all the initializer type configuration seems quite an undertaking for a very temporary win
(not to mention needing to come up with a solution to a couple of things that isn't "comment out that line from the relevant gem", which only really works for a local test :grin:)
Oleg Barenboim
@chessbyte
Oct 22 2015 18:16
not sure I agree that it is temporary
typical customer set up is to have a few appliances in a zone that run UI workers
the rest of the appliances are headless (ie. no UI)