These are chat archives for ManageIQ/manageiq/performance

19th
Feb 2016
Keenan Brock
@kbrock
Feb 19 2016 14:25
@Fryguy added # rows and collapsed recursive calls to better show ==> https://gist.github.com/kbrock/d3640de40b6b76f0ef27
note: ms- = ms without children
Jason Frey
@Fryguy
Feb 19 2016 14:31
So, I'm seeing that
Keenan Brock
@kbrock
Feb 19 2016 14:31
does that make sense at all?
I did need to add a few intermediate node to group code together
nodes, meaning methods
Oleg Barenboim
@chessbyte
Feb 19 2016 14:31
@kbrock what does the @ column mean?
Jason Frey
@Fryguy
Feb 19 2016 14:32
Is that what parent_arranged* means?
Because that method doesn't exist
Keenan Brock
@kbrock
Feb 19 2016 14:32
yea - it is arranged, but just the parent call
@chessbyte that is the offset from the beginning
@Fryguy would it be hard to rewrite prune_rbac to not be subtractive, but just run the actual query?
Oleg Barenboim
@chessbyte
Feb 19 2016 14:33
@kbrock I think I wrote prune_rbac
Keenan Brock
@kbrock
Feb 19 2016 14:33
aah
most of our queries in rbac say run this query - and tack on privs
Jason Frey
@Fryguy
Feb 19 2016 14:34
No, I wrote that
:)
Oleg Barenboim
@chessbyte
Feb 19 2016 14:34
fine
Keenan Brock
@kbrock
Feb 19 2016 14:34
that one says run rbac and take those away from main query

We're just fighting over who loves you more dad

you do

no you do
@simpsons

Jason Frey
@Fryguy
Feb 19 2016 14:34
Yeah, if you can get the same end results by rearranging go for it
Hahaha
The guts of the TreeBuilderVmsAndTemplates#relationship_tree can be changed as long as the inputs -> outputs is the same
In fact I had written some TODOs there a long time ago with ideas for making it better... Not sure you saw that in the code
Oleg Barenboim
@chessbyte
Feb 19 2016 14:36
high-level observation from your doc, @kbrock, is that it takes us ~15 seconds to get everything and ~2 seconds to render it
Keenan Brock
@kbrock
Feb 19 2016 14:37
yes
Oleg Barenboim
@chessbyte
Feb 19 2016 14:37
so, anything you can do to chop down the ~15 seconds will be worthwhile
hoping you can get that down to 3-4 seconds
Keenan Brock
@kbrock
Feb 19 2016 14:37
and that rendering would be less if we merged the features MR
Jason Frey
@Fryguy
Feb 19 2016 14:38
Right... One of the suggestions on the TODO is to get less data... I was investigating earlier to try to only get specific columns, but the Rails preloader doesn't play nice with polymorphic and taking columns for select
Oleg Barenboim
@chessbyte
Feb 19 2016 14:38
MR is downstream
Keenan Brock
@kbrock
Feb 19 2016 14:38
+1
one sec
Oh kbrock, the reason we fetch then filter is because it was hard to arrange the nodes after filtering, but if you can find a way...
Keenan Brock
@kbrock
Feb 19 2016 14:42
sweet - that gives me a lot. thanks @Fryguy
the bigger reason I want that is then we can convert rbac over to just scopes
Jared Deubel
@jdeubel
Feb 19 2016 16:02
@dmetzger57 hey
Dennis Metzger
@dmetzger57
Feb 19 2016 16:07
hi
Jared Deubel
@jdeubel
Feb 19 2016 16:08
hey I PM'ed you @dmetzger57
Dennis Metzger
@dmetzger57
Feb 19 2016 16:09
@jdeubel going private….
Keenan Brock
@kbrock
Feb 19 2016 21:25

Is there a better way to do a uniq like summary type method where you merge "dups"?

was 14 lines w/ flat_map. but I feel like it is still not quite right.
note: my nodes, equality, and summary functions are more complex

nodes = [{a: 1}, {a: 1}, {a: 1}, {b: 1}, {b: 1}]
rslt = nodes.group_by {|x| x.keys.first }.map { |n, v| {n => v.values.sum.sum }
# rslt => [{a: 3}, {b: 2}]
Jason Frey
@Fryguy
Feb 19 2016 21:27
Maybe?
nodes.each_with_object(Hash.new(0)) { |(k, v), h| h[k] += v }
Keenan Brock
@kbrock
Feb 19 2016 21:28
well, the output is an array of objects (not really hashes)
I used hashes to simplify - each of those nodes are SqlTimings / with child nodes that also need to be summarized...
I think I can work with that - thanks
nodes.each_with_object(Hash.new(0)) { |(k, v), h| h[k] += v }.values
I like java where you can delete while you are iterating over a collection
Jason Frey
@Fryguy
Feb 19 2016 21:31
group_by followed by .map or some kind of .transpose might be nicer though with objects
Keenan Brock
@kbrock
Feb 19 2016 21:31
transpose...
sweet
nodes.uniq { |n| n.keys.first }.map {|ns| summarize(n) }
that is pretty obvious
thanks
Jason Frey
@Fryguy
Feb 19 2016 21:35
doesn't the uniq {} lose records for you?
I thought you wanted to sum up across those with the same id
Keenan Brock
@kbrock
Feb 19 2016 21:36
yes it does
:(
Jason Frey
@Fryguy
Feb 19 2016 21:36
is your expected result:
nodes = [{a: 1}, {a: 1}, {a: 1}, {b: 1}, {b: 1}]
do_the_thing(nodes)
# => {:a => 3, :b => 2}
Keenan Brock
@kbrock
Feb 19 2016 21:36
yes
and summarize = another function. unique compares 2 different fields
      dups = []
      snodes = snodes.flat_map do |snode|
        if dups.empty? || (
          dups.first[:formatted_command_string] == snode[:formatted_command_string] &&
          dups.first[:params] == snode[:params]
        )
          dups << snode
          []
        else
          dups << snode
          summary = summarize_sql_nodes(dups)
          dups = []
          [summary]
        end
      end
      snodes << summarize_sql_nodes(dups) unless dups.empty?
      snodes
old
Jason Frey
@Fryguy
Feb 19 2016 21:40
Timings = Struct.new(:id, :time)
nodes = [Timings.new(:a, 1), Timings.new(:a, 1), Timings.new(:a, 1), Timings.new(:b, 1), Timings.new(:b, 1)]

nodes.group_by(&:id).map {|k, v| [k, v.map(&:time).sum]}.to_h
# => {:a=>3, :b=>2}

nodes.each_with_object(Hash.new(0)) {|n, h| h[n.id] += n.time}
# => {:a=>3, :b=>2}
the latter may be worse though if your function is oddly complex
Keenan Brock
@kbrock
Feb 19 2016 21:41
nah
    def summarize_sql_nodes(snodes)
      summary = snodes.first
      summary[:count] = snodes.size
      summary[:duration_milliseconds] = snodes.map { |s| s[:duration_milliseconds] }.sum
      summary
    end
It is too hard to create a sql node, so I munge it :(
Jason Frey
@Fryguy
Feb 19 2016 21:42
that was fun :D thanks!
Keenan Brock
@kbrock
Feb 19 2016 21:42
well - that was the summary method
I like the each with object - thanks. will look into that
ooh - output => [Timings, Timings]
Jason Frey
@Fryguy
Feb 19 2016 21:45
Ah, then the former might be better
nodes.group_by(&:id).values.map {|v| summarize(v)}
# => [Timings(time: 3), Timings(time: 2)]
lots of choices ;)
Keenan Brock
@kbrock
Feb 19 2016 21:47
ooh that values is sweet
I had it in the wrong spot
Jason Frey
@Fryguy
Feb 19 2016 21:48
yeah, you can use the &:method notation on objects, so code that looks otherwise ugly with hashes looks nice with objects
Keenan Brock
@kbrock
Feb 19 2016 22:39
@mathewd event "instantiation.active_record" gives you row_count for Vm.all.to_a
Is there something I can use with Vm.all.pluck(:id)?