These are chat archives for ManageIQ/manageiq/performance

21st
Nov 2016
Keenan Brock
@kbrock
Nov 21 2016 04:03

Huh. just in from /via ruby weekly

Rule 1: Let your Database do its Job
Let your database do what databases are good at, instead of doing it in Ruby.

Chris Arcand
@chrisarcand
Nov 21 2016 13:38
Amen.
Keenan Brock
@kbrock
Nov 21 2016 14:13
@NickLaMuro they say "never sort in ruby, always the database" - I know you and I will debate this further, but that is one more point pro sorting in db
Oleg Barenboim
@chessbyte
Nov 21 2016 14:40
nice article @kbrock
Keenan Brock
@kbrock
Nov 21 2016 14:40
fun to see someone else say this stuff more concisely than I can
Oleg Barenboim
@chessbyte
Nov 21 2016 14:41
next time someone asks what you are working on, just say "trying to make ManageIQ work as described in that article" :-)
Keenan Brock
@kbrock
Nov 21 2016 14:42
+1
lol
Ladislav Smola
@Ladas
Nov 21 2016 14:43
@kbrock ah, that looks cool, will make sure to read it in detail :-)
@kbrock ah, I didn't know rails can handle nested query that nicely
Ladislav Smola
@Ladas
Nov 21 2016 14:48
@kbrock I was always using to_sql
Keenan Brock
@kbrock
Nov 21 2016 15:40
fyi: x.save if x.changed? does seem necessary, but it makes a big difference. I've seen this in other places / projects as well. Lets not do this across the board, just putting out there that it can be necessary. dates, floats, and serialized columns seem to have the biggest problem.
example from ladas
Nick LaMuro
@NickLaMuro
Nov 21 2016 17:29
.merge is one of those super cool features of ActiveRelation that I have never had a chance to use effectively
Keenan Brock
@kbrock
Nov 21 2016 17:30
+1
@Ladas come to think of it. All the times I've introduced manually modifying changed? has been in the bulk loading code.

@Ladas @durandom in your code, you may want to play with

x.send("#{value}=",new_value) if x.send(value) != new_value

not sure if you have a lot of that code, but it is worth a quick hit to see if it saves that much time

Ladislav Smola
@Ladas
Nov 21 2016 18:15
@kbrock hm, I really rely on assign_attributes for this :-)
Keenan Brock
@kbrock
Nov 21 2016 18:16
ooh fun, attributes=(x) just calls that method
Ladislav Smola
@Ladas
Nov 21 2016 18:16
@kbrock x.save if x.changed? is huge, especially with 70k templates that does not really change :-)
@kbrock yees :-)
Keenan Brock
@kbrock
Nov 21 2016 18:16
right
wait
so the record knows it didn't change?
but it saves anyway?
not sure if an associated record changes - if it will detect it
Ladislav Smola
@Ladas
Nov 21 2016 18:17
@kbrock with save, it doesn't call the DB if !changed?
Keenan Brock
@kbrock
Nov 21 2016 18:17
ok
Ladislav Smola
@Ladas
Nov 21 2016 18:17
@kbrock but, it does some expensive computation
Keenan Brock
@kbrock
Nov 21 2016 18:17
ok
I noticed that for complex values (e.g. not integers / booleans), it doesn't always calculate changed correctly.
e.g. the nano seconds for a date will be different. so while technically they are different, the serialized value is the same
Ladislav Smola
@Ladas
Nov 21 2016 18:18
@kbrock although I am keen to see benchmark results from @durandom , he came with quite different number for memory consumption
@kbrock hm
Keenan Brock
@kbrock
Nov 21 2016 18:19
of course he did. I'm enjoying this feature progressing along
very fun to watch
Ladislav Smola
@Ladas
Nov 21 2016 18:19
hehe
Keenan Brock
@kbrock
Nov 21 2016 18:19
yea, so example. you have a serialized hash
and you store the value in the hash
top hash == {:x => "a"}
Ladislav Smola
@Ladas
Nov 21 2016 18:20
(for the associated) in DTO I save things flat :-)
Keenan Brock
@kbrock
Nov 21 2016 18:20
and def x=(val) ; top_hash[:x] = val ; end
Ladislav Smola
@Ladas
Nov 21 2016 18:20
except for the HABTM
Keenan Brock
@kbrock
Nov 21 2016 18:20
no
the objects themselves have serialization in them
time profiles
stuff like that
they have serialized data
if you call obj.x = "a" then it will think the serialized data has changed
it is probably a different hash
but it has the same value
but changed? will return false
recent rails may have fixed. but this came up a lot
Ladislav Smola
@Ladas
Nov 21 2016 18:21
hm
Keenan Brock
@kbrock
Nov 21 2016 18:21
and looking at the way we store data, we do funky stuff. so it may make a difference
Ladislav Smola
@Ladas
Nov 21 2016 18:21
so should I make the '.changed?' optional?
Keenan Brock
@kbrock
Nov 21 2016 18:22
I'm wondering if you should implement assign_attributes
Ladislav Smola
@Ladas
Nov 21 2016 18:22
I was thinking to close this under dto_memory_optimized, dto_time_optimized flags in settings :-)
Keenan Brock
@kbrock
Nov 21 2016 18:22
hmm
Ladislav Smola
@Ladas
Nov 21 2016 18:23
hmm
Nick LaMuro
@NickLaMuro
Nov 21 2016 18:23
hmm?
Ladislav Smola
@Ladas
Nov 21 2016 18:23
well I guess as a next step :-)
but so far assign_attributes worked well
but we might see failures with some data in some providers
then..
but I would like to be able to switch these optimizations on and off
Keenan Brock
@kbrock
Nov 21 2016 18:36
#define FAST true
Ladislav Smola
@Ladas
Nov 21 2016 18:43
:-)