These are chat archives for ManageIQ/manageiq/performance

31st
Mar 2016
Jason Frey
@Fryguy
Mar 31 2016 14:34
@kbrock Reading back
Keenan Brock
@kbrock
Mar 31 2016 14:36
@Fryguy I'm having trouble seeing the code running in MiqEmsMetricsProcessorWorker
it is basically empty
Jason Frey
@Fryguy
Mar 31 2016 14:46

:point_up: March 30, 2016 5:03 PM

In general, if you know your callers will use all columns of the index, you want the order of the index columns to be such that the first ones have the higest cardinality (i.e. will narrow down the number of rows the faster)

Keenan Brock
@kbrock
Mar 31 2016 14:47
100%
Jason Frey
@Fryguy
Mar 31 2016 14:47
in reality it probably saves like 0.00000001 seconds :D
Keenan Brock
@kbrock
Mar 31 2016 14:47
but if you do resource_type='A' and resource_id in (1,2,3,4,5,6,7)
agreed that the beginning should be more diverse. much better
just proposed that over in rubygems.org ;)
Jason Frey
@Fryguy
Mar 31 2016 14:48
yeah, I'm not sure with IN clause
depends what we do more often and on what table
Keenan Brock
@kbrock
Mar 31 2016 14:48
[----] D, [2016-03-30T03:40:22.656778 #19036:463994] DEBUG -- :   VimPerformanceState Load (1473229.6ms)
SELECT "vim_performance_states".*
FROM "vim_performance_states"
WHERE "vim_performance_states"."resource_type" = 'VmOrTemplate'
AND "vim_performance_states"."resource_id" IN (1000000000020, 1000000000038, 1000000000092, 1000000000100, 1000000000118, 1000000000130,
Jason Frey
@Fryguy
Mar 31 2016 14:49
Regardin specifying the :btree and :name...
we don't specify btree...that's the Rails default (or perhaps postgres default)
Keenan Brock
@kbrock
Mar 31 2016 14:49
cool - thought so
list of ids is LONGish, but (1,473,229.6ms)
Jason Frey
@Fryguy
Mar 31 2016 14:49
we used to only specify name when the autogenerated name was too long
but now, I'm not sure what we've decided...I usually red flag a PR if I see the :name there and tell the author not to do that
Keenan Brock
@kbrock
Mar 31 2016 14:50
turns out per @matthewd the naming convention for indexes changed across rails versions, so the older ones needed to be specificed
Jason Frey
@Fryguy
Mar 31 2016 14:50
Did we go back and change them?
Keenan Brock
@kbrock
Mar 31 2016 14:50
dunno, but the schema.rb is different
Matthew Draper
@matthewd
Mar 31 2016 14:50
Yes
Jason Frey
@Fryguy
Mar 31 2016 14:50
Ah ok...I forgot about that, then
so, should we specify name, always, moving forward?
Keenan Brock
@kbrock
Mar 31 2016 14:50
heh. and you asked is matthew on the ball...
no
not forward
backwards
Jason Frey
@Fryguy
Mar 31 2016 14:50
ok
Matthew Draper
@matthewd
Mar 31 2016 14:51
It was part of the Rails 4 changes
Keenan Brock
@kbrock
Mar 31 2016 14:51
i combed through EVERY line of that PR and missed that ;)
Matthew Draper
@matthewd
Mar 31 2016 14:51
(and the solution to masssssive ID lists isn't to make them slightly faster by rearranging indexes.. it's to stop doing that ;))
Keenan Brock
@kbrock
Mar 31 2016 14:52
lol
but I like them... especially in RBAC :)
but to be honest
if we use a join, inner select or a list - we need it to be indexed properly
Jason Frey
@Fryguy
Mar 31 2016 14:53
all of those should work fine with the resource_id, resource_type as is I would think
Matthew Draper
@matthewd
Mar 31 2016 14:54
Yeah, given a long somewhat varied list, the page locality of having the class first is going to be minimal benefit
Keenan Brock
@kbrock
Mar 31 2016 14:54
thanks - will look into the columns brought back
Matthew Draper
@matthewd
Mar 31 2016 14:55
Especially given there are only a handful of possible classes
Keenan Brock
@kbrock
Mar 31 2016 14:55
dumb question - how do I find out where this code is called? I looked in MiqEmsMetricsProcessorWorker and there isn't really any code there
Matthew Draper
@matthewd
Mar 31 2016 14:55
(My theory that "don't do that" would help is that the ID list came from somewhere, and generally that somewhere is a more sane filtering on the same table)
Jason Frey
@Fryguy
Mar 31 2016 14:56
@kbrock you mean that states query?
Keenan Brock
@kbrock
Mar 31 2016 14:56
yes
I want to know the code that calls that
a) use a subquery instead of an id list
b) use a select()
that whole file really
Keenan Brock
@kbrock
Mar 31 2016 14:58
yes, curious the workflow that calls it
but I see that referenced in only 1 place
I just looked at the workers - and they are empty
so it is tricky to figure out where the actual work is declared
Jason Frey
@Fryguy
Mar 31 2016 14:58
probably dyanimically called
Keenan Brock
@kbrock
Mar 31 2016 15:00
our use of obj.send("#{x}_y") and get_const("#{x.upcase}_Y") makes the code very difficult to grep
Jason Frey
@Fryguy
Mar 31 2016 15:00
I found it pretty quick ;)
Keenan Brock
@kbrock
Mar 31 2016 15:01
that is not the query
that query has timestamps in it
the other query only has ids
Jason Frey
@Fryguy
Mar 31 2016 15:02
ah ok
Keenan Brock
@kbrock
Mar 31 2016 15:02
maybe that is the query that fetches the objects. converts into ids and then fetches again
Jason Frey
@Fryguy
Mar 31 2016 15:02
it really shouldn't, but maybe something isn't working as expected
Keenan Brock
@kbrock
Mar 31 2016 15:02
a. I don't know how workers decide the job to run. the worker seems to have nothing in it
kidding? we do that all over the place. especially in rbac - we sometimes do that like 3 times
Jason Frey
@Fryguy
Mar 31 2016 15:04
well, in RBAC, yes, but this isn't RBAC...in the C&U stuff for vim_performance_state, we should be preloading them, and then looking them up
at least that's how I remember it...maybe it's been changed

I don't know how workers decide the job to run

It's a queue item, so it runs what it's told

Keenan Brock
@kbrock
Mar 31 2016 15:05
so we don't really need workers / runners?
for the most part?
cool - then I'll look at the job and ignore the worker. thanks
Jason Frey
@Fryguy
Mar 31 2016 15:06
yeah, focus on the job itself
technically everything can run on the generic queue worker :)
in fact it used to
Keenan Brock
@kbrock
Mar 31 2016 15:08
I'm sure you are setting it up to just name queues and use the generic worker code behind it
Jason Frey
@Fryguy
Mar 31 2016 15:15
yeah, they all derive from QueueWorkerBase, which is the brunt of the work
Keenan Brock
@kbrock
Mar 31 2016 15:20
wow ARGF is pretty neat 2.3.0 ref better docs in older ref
Jason Frey
@Fryguy
Mar 31 2016 15:23
@kbrock I can't find that query either..at least not without a qualifying timestamp
the only thing I can think of is automate
Can you get more log context? The line they give before it is not the same process, so I think it's unrelated
I think line 247 there will break the preloading that's done on line 229
it's basically preloading twice, which is weird
Keenan Brock
@kbrock
Mar 31 2016 16:39
that looks like a bug
so you think that query is part of a preload?
probably
thanks
Jason Frey
@Fryguy
Mar 31 2016 16:39
yeah, because preload works by sending a bunch of ids
not sure...hope that helps...good luck :four_leaf_clover:
Keenan Brock
@kbrock
Mar 31 2016 16:40
thanks
thanks for the help. think I'll run with that
Jason Frey
@Fryguy
Mar 31 2016 16:41
yeah, my gut is telling me preload, I'm just not sure which one :/
Keenan Brock
@kbrock
Mar 31 2016 16:41
:) yea - there is a condition on that. I'll have to play with preload and see where it takes me