These are chat archives for ManageIQ/manageiq/performance

4th
Mar 2016
Keenan Brock
@kbrock
Mar 04 2016 00:01
aah, something with Relationship.arranged_rels_to_resources
Keenan Brock
@kbrock
Mar 04 2016 00:16
@Fryguy do you know how to make this work if you don't have depth? here
Keenan Brock
@kbrock
Mar 04 2016 03:55
got it - nvr mind
Keenan Brock
@kbrock
Mar 04 2016 06:21
For "My Services" the current numbers are:
9710 => 187 queries.
19k => 9.7k rows
68s => 9.8s ( ~1/7 the time )
need to run tests and make sure I didn't botch something up
vmdb :( tests failed... here we go
Alex Krzos
@akrzos
Mar 04 2016 13:30
wow 1/7th
Keenan Brock
@kbrock
Mar 04 2016 13:43
still have 1 test failing. I may have lost some stuff there
there is 1 or two rbac changes that will probably effect everyone (in a good way) but yea. reading that line, it could be in a bad way too :)
sure wish I was on master or 5.5 - this stuff is so different between 5.4 and master
@matthewd ping if you have any spare sanity check cycles
Keenan Brock
@kbrock
Mar 04 2016 14:07
@fryguy I thought we wrote our own acts_as_tree like plugin but can't find it. pointers?
Jason Frey
@Fryguy
Mar 04 2016 15:21
we did not
Keenan Brock
@kbrock
Mar 04 2016 15:21
thanks
Jason Frey
@Fryguy
Mar 04 2016 15:21
maybe you're thinking of the relationships table?
Keenan Brock
@kbrock
Mar 04 2016 15:21
must be
Jason Frey
@Fryguy
Mar 04 2016 15:21
and Relationship model?
Keenan Brock
@kbrock
Mar 04 2016 15:21
yea - at one time I thought it was a rewrite
and we also use act_as_tree
I just noticed we have "parent_id" for services
(technically service_id)
so we are recursing
I have a way of getting in 1 fell swoop - but feels like I'm rewriting act_as_tree/ancestry, just using parent_id concept
performance numbers are drastically different - feel like I'm doing your TreeBuilderVmTemplate change w/ services (load all things)
@Fryguy thanks - that must be what I'm remembering
Jason Frey
@Fryguy
Mar 04 2016 15:26
yeah, we have a few places that still have parent_id because they are really old :older_man:
snapshots is another place
Keenan Brock
@kbrock
Mar 04 2016 15:27
thnx. good to remember
Jason Frey
@Fryguy
Mar 04 2016 15:27
I wanted to switch them all to ancestry but never had the time
Keenan Brock
@kbrock
Mar 04 2016 15:27
+1
Oleg Barenboim
@chessbyte
Mar 04 2016 15:27
the time has come, the Walrus said ...
Jason Frey
@Fryguy
Mar 04 2016 15:27
in before @matthewd says "closure_tree"
Matthew Draper
@matthewd
Mar 04 2016 15:28
Too quick for me :)
Keenan Brock
@kbrock
Mar 04 2016 15:28
:)
matthewd @matthewd was actually trying to remember what it was called
Keenan Brock
@kbrock
Mar 04 2016 15:28
lol
Jason Frey
@Fryguy
Mar 04 2016 15:28
I was thikning about closure tree...isn't it awfully expensive for really large trees?
because it keeps like a cartesian product of all pairs
or am I just not understanding how it works
Matthew Draper
@matthewd
Mar 04 2016 15:29
All pairs of ancestors/descendants
So it's expensive for very deep trees
Jason Frey
@Fryguy
Mar 04 2016 15:29
ah right..that's what i meant
I feel like ancestry has that nice balance of not storing a zillion rows but still allowing you to do single query trees
though I agree that an array column vs a string column might be more performant
Matthew Draper
@matthewd
Mar 04 2016 15:31
I mostly like the way that closure_tree is built entirely around AR relations, tbh
Jason Frey
@Fryguy
Mar 04 2016 15:31
ancestry is too
maybe not as nice though :)
Matthew Draper
@matthewd
Mar 04 2016 15:32
The fact it's performantly optimal for certain operations like subtree relocation is conceptually nice, but unlikely to be something we actually care about in practice
Jason Frey
@Fryguy
Mar 04 2016 15:32
yeah, I don't think we do that much at all
Matthew Draper
@matthewd
Mar 04 2016 15:32
But, e.g., the ability to find all ancestors and descendants of a given node in a single query, without the cost of %,1234,% does seem valuable
Keenan Brock
@kbrock
Mar 04 2016 15:34
there are other ways like arrays and ltree. but closure trees look pretty cool
can I introduce a schma migration to casablanca? ;)
Joe Rafaniello
@jrafanie
Mar 04 2016 15:34
capablanca?
:laughing:
Keenan Brock
@kbrock
Mar 04 2016 15:35
auto correct :(
ignore - was snark anyway
whoa...I did not know slideshare embedded in gitter :P
anyway, I used that a lot as a reference when I chose ancestry initially
Matthew Draper
@matthewd
Mar 04 2016 15:37
I used to use nested sets, which is pretty storage optimal, but needs rather unpleasant bulk updates for some operations
Keenan Brock
@kbrock
Mar 04 2016 15:43
yea - I like that slide deck, but they don't do arrays. which seems just as efficient as closures (+/- 1 query) - very similar to path enumeration, but without the likes
right now for services, I'm looking at slide 17 - SQL-99 recursive syntax
Matthew Draper
@matthewd
Mar 04 2016 15:44
Arrays and strings are conceptually the same
Keenan Brock
@kbrock
Mar 04 2016 15:44
+1
Matthew Draper
@matthewd
Mar 04 2016 15:44
Value-in-array and string-like are the same fundamental complexity
Jason Frey
@Fryguy
Mar 04 2016 15:45
@matthewd I thought Arrays could index a little more efficiently
Keenan Brock
@kbrock
Mar 04 2016 15:45
except string/arrays is easy to query children (I think)
Matthew Draper
@matthewd
Mar 04 2016 15:46
@Fryguy more easily, perhaps — you could technically create a similarly viable index for the like query
Jason Frey
@Fryguy
Mar 04 2016 15:46
Either way I never had a problem with LIKE personally...Btrees are pretty damn efficient ¯\_(ツ)_/¯
Matthew Draper
@matthewd
Mar 04 2016 15:47
If you're doing a like with a leading %, you're not using an index at all (unless you have the magic index thing)
Jason Frey
@Fryguy
Mar 04 2016 15:47
yes
luckily ancestry doesn't do that :)
Keenan Brock
@kbrock
Mar 04 2016 15:47
well, if you are using utf, like with '%' at the end doesn't use the index much either
Matthew Draper
@matthewd
Mar 04 2016 15:47
But if you index an array in a way that you can arbitrarily query its contents, you're creating exactly the same data structure as closure_tree anyway
Jason Frey
@Fryguy
Mar 04 2016 15:47
as long as you follow the left-to-right thing you're good
Matthew Draper
@matthewd
Mar 04 2016 15:48
Right.. which means you can't ascend the tree in a single query = no relation
Keenan Brock
@kbrock
Mar 04 2016 15:48
@Fryguy if you are using c locale - I agree. I'm still not convinced with our locale that we are (or was collation the word)
Jason Frey
@Fryguy
Mar 04 2016 15:49
@matthewd I don't follow...getting the ancestors of a tree is a single query
with index
Keenan Brock
@kbrock
Mar 04 2016 15:49
@matthewd you can say that the array contains the parent ids - so you get all children
heck, the array contains the node's id
Jason Frey
@Fryguy
Mar 04 2016 15:49
@kbrock I don't understand...we use UTF8 collation
Keenan Brock
@kbrock
Mar 04 2016 15:49
+1
Jason Frey
@Fryguy
Mar 04 2016 15:50
if we're not we might have other problems :laughing:
Keenan Brock
@kbrock
Mar 04 2016 15:50
and as I've said before, I've seen strange ancestry explain plans to suggest we're not using the index
Matthew Draper
@matthewd
Mar 04 2016 15:50
LIKE and UTF8 don't get along very well
Keenan Brock
@kbrock
Mar 04 2016 15:50
+1
Jason Frey
@Fryguy
Mar 04 2016 15:50
really? I did not know that
Matthew Draper
@matthewd
Mar 04 2016 15:50
Because of the complexity of Unicode sorting rules
Jason Frey
@Fryguy
Mar 04 2016 15:50
@kbrock if you have examples that you can share...would love to see
Keenan Brock
@kbrock
Mar 04 2016 15:50
heh
THIS is why I got into the ancestry gem in the first place
because of the like '/1/%' and UTF8 indexes
ok
Matthew Draper
@matthewd
Mar 04 2016 15:51
.. but that just means the ancestry string should be using C collation
Jason Frey
@Fryguy
Mar 04 2016 15:51
need numbers :D
Matthew Draper
@matthewd
Mar 04 2016 15:51
There's no reason to leave it at the default
Jason Frey
@Fryguy
Mar 04 2016 15:51
yeah, we can tweak that one column if necessary
Keenan Brock
@kbrock
Mar 04 2016 15:51
matthewd in our current setup, we can only change the collation for the whole database
^ I think
there is an extension, but not part of core yet (e.g. json, hstore, uuid are extension but part of core)
Jason Frey
@Fryguy
Mar 04 2016 15:53
or we can just send an ALTER TABLE, right?
Matthew Draper
@matthewd
Mar 04 2016 15:53
The collation feature allows specifying the sort order and character classification behavior of data per-column, or even per-operation.
CREATE TABLE test1 (
    a text COLLATE "de_DE",
    b text COLLATE "es_ES",
    ...
);
Keenan Brock
@kbrock
Mar 04 2016 15:53
is that in core or an extension?
link?
very cool
Keenan Brock
@kbrock
Mar 04 2016 15:54
9.1 ! nice
I was looking into a case insensitive collation for our columns, so we could get rid of our downcase(col) all over the place
tangent
matthewd
Jason Frey
@Fryguy
Mar 04 2016 15:54
so was @yrudman
Keenan Brock
@kbrock
Mar 04 2016 15:55
ran into a nuance
relation << "a" << "b" ... relation.size == 2, relation.count == 0
Matthew Draper
@matthewd
Mar 04 2016 15:55
.. is that new information? :confused:
Keenan Brock
@kbrock
Mar 04 2016 15:56
well. I knew it COULD happen
just got hit by it thought :)
Keenan Brock
@kbrock
Mar 04 2016 17:18
@Fryguy do you know the back story behind ManageIQ/manageiq@9357d60
I was using options to cache the full tree, not sure why it has been removed. can I just add it back?
what do we buy making that method so specialized and only passing in count_only vs options?
Keenan Brock
@kbrock
Mar 04 2016 19:11
@Fryguy any suggestions on the return value for TreeBuilderServices < FullTreeBuilder - I thought I mimicked the output from TreeBuilderVmAndTemplate - but no idea where to put node prefixes and stuff like that
Jason Frey
@Fryguy
Mar 04 2016 19:12
FreeTreeBuilder#tree returns the JSON for the UI
as long as YourTreeSubclass#relationship_tree returns what is structured like arranged_nodes
Keenan Brock
@kbrock
Mar 04 2016 19:13
I implemented relationship_tree
is it 100% hashes
or is it hashes + arrays?
Jason Frey
@Fryguy
Mar 04 2016 19:13
make it look like an ancestry arranged_nodes and it will just work
Keenan Brock
@kbrock
Mar 04 2016 19:13
ok
thought I did that
thanks
to get the details for a node, it uses x_build_single_node, so if you're refactoring I would think that should Just Work™
Keenan Brock
@kbrock
Mar 04 2016 19:51
@akrzos do you have a database with a lot of services in them? don't want to upgrade the one I have from 5.4 to HEAD
Keenan Brock
@kbrock
Mar 04 2016 20:35
@Fryguy non-vpn link: ManageIQ/manageiq#7114
working towards getting that for a full tree builder
need to get a patch out as soon as I can
Jason Frey
@Fryguy
Mar 04 2016 20:45
Is that WIP?
Keenan Brock
@kbrock
Mar 04 2016 20:45
I'm planning on giving a similar patch to a customer for that... so kinda/kinda not
Jason Frey
@Fryguy
Mar 04 2016 20:46
ok, I ask because there were some comments and also some empty testing paths
Keenan Brock
@kbrock
Mar 04 2016 20:46
aah
Jason Frey
@Fryguy
Mar 04 2016 20:46
ok, just one :)
Keenan Brock
@kbrock
Mar 04 2016 20:46
I'm living in 2 branches
Matthew Draper
@matthewd
Mar 04 2016 20:47
Am I allowed to pick on points of modernity, or will you just complain that makes it harder to backport? :)
The includes/extends in this mixin immediately jump out at me
Keenan Brock
@kbrock
Mar 04 2016 20:47
yea
pick away
Matthew Draper
@matthewd
Mar 04 2016 20:47
But I'd believe you if you claimed older versions + concerns were a no-go
Keenan Brock
@kbrock
Mar 04 2016 20:47
um, think concerns are good
but I have a variable in there
so didn't know how to mix in a concern and get an act_as_ :var1 in there
ruby 2.0, rails 4.2(?)3.2
Keenan Brock
@kbrock
Mar 04 2016 20:55
@Fryguy in ancestry - why did you use an ordered hash?
Jason Frey
@Fryguy
Mar 04 2016 20:56
because the code for ancestry runs on Ruby 1.8
Keenan Brock
@kbrock
Mar 04 2016 20:56
thanks
Keenan Brock
@kbrock
Mar 04 2016 21:21
@akrzos you around?
Alex Krzos
@akrzos
Mar 04 2016 22:22
My bad, here now
@kbrock
Keenan Brock
@kbrock
Mar 04 2016 22:22
cool
I sent out an email with a pach (5 seconds ago)
think I got services improved. onto next one
happy weekend
:)
Alex Krzos
@akrzos
Mar 04 2016 22:23
gotcha, do you need this validated?
I have a 5.4.5.0 appliance with the db on it I can patch with 835