These are chat archives for ManageIQ/manageiq/performance

30th
Nov 2015
shows how to use flame graphs to better visualize where resources (memory, time, cpu, ...) is used - but with the rollups allows a quick hit
Alex Krzos
@akrzos
Nov 30 2015 13:21
:+1: Brendan Gregg
Oleg Barenboim
@chessbyte
Nov 30 2015 13:22
@akrzos what is that +1 in reference to?
Alex Krzos
@akrzos
Nov 30 2015 13:22
@chessbyte kbrock posting Brendan Gregg's Flame Graph Performance Talk
Oleg Barenboim
@chessbyte
Nov 30 2015 13:28
thanks
Keenan Brock
@kbrock
Nov 30 2015 18:34
@dmetzger57 ok, found a good one, but not sure if this is too intrusive
for cap and U, we insert a record into the queue if it doesn't exist
but, "if it exists" requires that we bring back all rows, and look at the serialized attributes to see if it is the record we care about.
if we changed the methods from perf_capture(misc, 'weekly') to perf_capture_weekly(misc) we would not need to serialize all that data just for an exists
and we would be one step closer to simplifying our "envelope" (currently, we use too many fields to determine uniqueness, destination)
Keenan Brock
@kbrock
Nov 30 2015 18:39
why I care?
perf_capture of 10vms takes 73 queries, 5mb / 64k objects
inserting into the queue takes 58 queries, 2.5mb / 35k objects
oh, and in that sample, the job is already in the queue, so that is 100% detection, 0 inserts
Keenan Brock
@kbrock
Nov 30 2015 19:47
@dmetzger57 I want to back patch ManageIQ/manageiq#5610
on http://localhost:3000/vm_infra/explorer (12 VMs )- it is causing 60+ queries
Dennis Metzger
@dmetzger57
Nov 30 2015 19:50
is this related to 1281999?
Keenan Brock
@kbrock
Nov 30 2015 19:51
I'm on that page
and 75% of the queries are feature privilidge lookup
I could dive more - but don't want to bother
let me know if you need more data
at first I wanted in master
now, I want in every branch
I'm diving into rbac
our rbac needs some love for that page
looks like we are making 4 of the same query to bring back all ids / vms from the table
so while the sorting / bringing back all rows is bad, having to do it 4 times is exacerbating it
Joe Rafaniello
@jrafanie
Nov 30 2015 21:23
@akrzos I looked at memoist some more (one of the libraries we use alot in reports)
I have "Folder to VMs Relationships" down from 46 seconds to 43 seconds with my first PR: matthewrudy/memoist#36
I went back to it and found alot more potential
Keenan Brock
@kbrock
Nov 30 2015 21:25
@jrafanie you're lucky
I'm in RBAC land
Joe Rafaniello
@jrafanie
Nov 30 2015 21:25
got it down to 18.5 seconds
Alex Krzos
@akrzos
Nov 30 2015 21:25
@jrafanie Do you know if that will make it into 5.5?
Keenan Brock
@kbrock
Nov 30 2015 21:25
we're bringing back the VMs (or counts) like 6 times
Joe Rafaniello
@jrafanie
Nov 30 2015 21:25
@akrzos probably not, it's all in memoist gem
Keenan Brock
@kbrock
Nov 30 2015 21:26
@jrafanie could you take a peek at https://github.com/ManageIQ/manageiq/pull/5610#issuecomment-160743192 - I really want this into 5.5
Joe Rafaniello
@jrafanie
Nov 30 2015 21:26
so we'd have to get a release with my second pr in
sorry, @kbrock been heads down on this thing... yeah, I'll look now, anythign else urgent?
it's merged
Oleg merged it
Keenan Brock
@kbrock
Nov 30 2015 21:27
ooh - thanks chess guy
Alex Krzos
@akrzos
Nov 30 2015 21:27
@jrafanie gotcha, so basically would have to wait until memoist gem is built with the fix and if we use that version in miq
Joe Rafaniello
@jrafanie
Nov 30 2015 21:27
yes @akrzos, the person has been very quick in the past.. he merged my other PR quickly (days)
note, my change (for reports using our relationships mixin) increases memory usage but decreases report times and total object allocations
we might be able to flush the caches after the report is done instead of constantly while building the report
Joe Rafaniello
@jrafanie
Nov 30 2015 21:34
# Using existing memoist 0.11.0
Time elapsed |      RSS change |       Total RSS | Allocated Objects |                         Description
 24.192566 |       118341632 |       303005696 |        16390381 |           33 - Host Summary for VMs
 19.924119 |        19668992 |       322674688 |        12986524 |               88 - VM Relationships
 46.289293 |         5808128 |       328482816 |        28923805 |    89 - Folder to VMs Relationships

# Using memoist with PR 36 to decrease allocations
Time elapsed |      RSS change |       Total RSS | Allocated Objects |                         Description
 22.218585 |       116183040 |       309014528 |        13301716 |           33 - Host Summary for VMs
 18.778133 |        16748544 |       325771264 |        11208700 |               88 - VM Relationships
 43.014996 |        12566528 |       338341888 |        25363805 |    89 - Folder to VMs Relationships

# Using latest change: https://github.com/matthewrudy/memoist/compare/master...jrafanie:faster2
Time elapsed |      RSS change |       Total RSS | Allocated Objects |                         Description
 14.720751 |       142651392 |       337928192 |         9179253 |           33 - Host Summary for VMs
 11.804993 |        44146688 |       382074880 |         7085406 |               88 - VM Relationships
 18.641427 |        22925312 |       405004288 |        10977805 |    89 - Folder to VMs Relationships
@akrzos ^
29 million allocations -> 11 million :fire: :fire: :fire:
Alex Krzos
@akrzos
Nov 30 2015 21:39
Interesting, we will have to see how this affects performance when the scale is ramped up
Joe Rafaniello
@jrafanie
Nov 30 2015 21:40
Yeah, I'd be curious about the memory increase to see if we need a "flush caches created during a report" that happens at the end of report generation
the most important change is that flush_cache with no specific methods (flush all caches for the current object/class) is much faster because we kept asking the same question and getting the same answer: what instance variable should I use to store this memoized value... and what's the unmemoized method name
flush_cache now:                      68540.6 i/s
flush_cache after PR 34:              16082.1 i/s - 4.26x slower
flush_cache after PR 36:               3844.2 i/s - 17.83x slower
Keenan Brock
@kbrock
Nov 30 2015 21:55
@jrafanie have you used web-console gem on manageiq?
Joe Rafaniello
@jrafanie
Nov 30 2015 21:59
@kbrock it doesn't sound familiar
Keenan Brock
@kbrock
Nov 30 2015 22:05
it is in rails by default now. shows a debug console in the web
hmm, I'll test if byebug works in the middle of requests
Keenan Brock
@kbrock
Nov 30 2015 22:56
@matthewd @jrafanie did we modifyjavascript_include_tag?
In development, >50% of our render time is spent in there.
Matthew Draper
@matthewd
Nov 30 2015 22:57
aka "in sprockets"
Keenan Brock
@kbrock
Nov 30 2015 22:57
aah - yes, most of that is in application.js which is big
Joe Rafaniello
@jrafanie
Nov 30 2015 22:57
I'm not familiar with it @kbrock
Keenan Brock
@kbrock
Nov 30 2015 22:58
over 50% of my page render time is in that one file
Joe Rafaniello
@jrafanie
Nov 30 2015 22:59
wait for http2 ;-)
Keenan Brock
@kbrock
Nov 30 2015 22:59
heh - nice
Matthew Draper
@matthewd
Nov 30 2015 22:59
@kbrock you might want to look into what discourse do there
Keenan Brock
@kbrock
Nov 30 2015 22:59
thnx
Matthew Draper
@matthewd
Nov 30 2015 22:59
.. but I think it's basically "avoid sprockets"
Keenan Brock
@kbrock
Nov 30 2015 22:59
next question for you @matthewd
lol
Matthew Draper
@matthewd
Nov 30 2015 22:59
Sam wrote a blog post about it, IIRC
Keenan Brock
@kbrock
Nov 30 2015 22:59
rbac.rb:320 if objects.present?
thanks - will do
if that is an array, present? is good. If that is an association, hit accesses the database
we have a couple of checks around, that should "speed up" our processing, but I think it just adds to it
thoughts on removing a few of those, or ways to change them to not hit db for associations/class?
Matthew Draper
@matthewd
Nov 30 2015 23:01
Ah, possibly… that would be new in rails4
Keenan Brock
@kbrock
Nov 30 2015 23:02
was going to change to if object.nil? || object.kind_of?(Array) && !object.present?
but mixed on that
there are a few empty? that I'll have to change to present?
Matthew Draper
@matthewd
Nov 30 2015 23:03
ughhh
This looks like an optimisation, but without the condition, the behaviour would totally change
Keenan Brock
@kbrock
Nov 30 2015 23:04
yes
for vms page - there are about 12 queries that are fetching vms / vms.count
want to get down to 3
granted - I wish we didn't bring back records for ALL VMs, but not doing it multiple times may be a first step
Matthew Draper
@matthewd
Nov 30 2015 23:05
Considering what search ends up doing, maybe simplest to objects = objects.to_a for now?
Keenan Brock
@kbrock
Nov 30 2015 23:06
yea - :( - wanted to try and get the sql more into rbac rather than further away
but rbac then coverts to ids and then back to objects again :(
great idea - will try to_a
Matthew Draper
@matthewd
Nov 30 2015 23:11
    if targets.kind_of?(Array) && targets.first.kind_of?(Numeric)
      target_ids       = targets
    else
      target_ids       = targets.respond_to?(:ids) ? targets.ids : targets.collect(&:id)
      unless target_ids.empty?
        klass            = targets.respond_to?(:klass) ? klass : targets.first.class.base_class unless klass.respond_to?(:find)
        results_format ||= :objects
      end
    end

    if target_ids.empty?
      target_ids = nil
    else
      ids_clause = ["#{klass.table_name}.id IN (?)", target_ids] if klass.respond_to?(:table_name)
    end
@kbrock that might avoid queries other than SELECT id for targets
@kbrock but I think the best simple change to make right now, would be to teach search the difference between empty targets and nil targets
Matthew Draper
@matthewd
Nov 30 2015 23:17
nil / option not supplied == do not filter; empty list should mean it returns an empty list
Immediate change there would be to just deprecate the ability to pass an empty list