These are chat archives for ManageIQ/manageiq/performance

12th
Nov 2015
Alex Krzos
@akrzos
Nov 12 2015 18:01
perf meeting if you are available https://bluejeans.com/471610744/
@kbrock @jrafanie @Fryguy ^
Keenan Brock
@kbrock
Nov 12 2015 19:04
@matthewd / @jrafanie more fun miq_queue goodness - we spend just as much time sending transaction "BEGIN", "COMMIT" as we spend in "INSERT INTO miq_queue"
@dmetzger57 I found my issue. looks like our query_counter class didn't properly handle schema lines. so it was showing a much higher count than approperiate. Now it is much more consistent for me
@matthewd is there a way to know how many times a sql query is NOT run - because active record says "already run, here is the cached results"?
I had thought query_counter got those, but looks like it does not.
Matthew Draper
@matthewd
Nov 12 2015 19:08
They certainly get logged, so.. find where that happens, and work backwards?
Does seem like something you'd like to know
Jason Frey
@Fryguy
Nov 12 2015 19:10
I have an open issue on Rails for BEGIN/COMMIT when it's useless
but never got a chance to code it up myself
rails/rails#17937
In our old version when we had it monkey patched locally, it gave significant timing improvements, particularly in EmsRefresh
@kbrock PR please for your QueryCounter find?
Keenan Brock
@kbrock
Nov 12 2015 19:12
I'm noticing for 1000 miq queue inserts, it takes:
[code: 1639ms, sql[1000]: 269ms othr[2000]: 244ms] +obj: 1_024_061 +mem: 7_285_028
[mGC +live: 1_024_060]
1.6 seconds in code
270ms in sql
244ms in begin/commit
that seems pretty steep
Jason Frey
@Fryguy
Nov 12 2015 19:13
seems about right
Keenan Brock
@kbrock
Nov 12 2015 19:13
the insert is the same cost as the begin/end ?
so basically it doubled that?
Jason Frey
@Fryguy
Nov 12 2015 19:13
sure because it's all about the round trip
Keenan Brock
@kbrock
Nov 12 2015 19:13
ooh
yes
Jason Frey
@Fryguy
Nov 12 2015 19:13
the latency, if you will
Keenan Brock
@kbrock
Nov 12 2015 19:14
BUT, feels like it isn't necessary
Jason Frey
@Fryguy
Nov 12 2015 19:14
the actual time in PG is nothing
Keenan Brock
@kbrock
Nov 12 2015 19:14
yes
Matthew Draper
@matthewd
Nov 12 2015 19:14
By those numbers the insert is twice as expensive
Keenan Brock
@kbrock
Nov 12 2015 19:14
yes
which makes sense as well.
Jason Frey
@Fryguy
Nov 12 2015 19:14
well, a BEGIN is much less expensive on the PG side as the actual insert
Matthew Draper
@matthewd
Nov 12 2015 19:14
But yeah, it's not that the begin/commit is expensive, so much as that the insert is cheap
Keenan Brock
@kbrock
Nov 12 2015 19:14
lol
Jason Frey
@Fryguy
Nov 12 2015 19:14
right
Keenan Brock
@kbrock
Nov 12 2015 19:15
would be nice if the transaction were not issued
Jason Frey
@Fryguy
Nov 12 2015 19:15
techcanilly, you probably pay all the penalty on COMMIT
BEGIN is likely nothing
you can't not use transactions
Keenan Brock
@kbrock
Nov 12 2015 19:15
hmm
Matthew Draper
@matthewd
Nov 12 2015 19:15
Indeed; begin is little more than a lock to allocate an xid
Jason Frey
@Fryguy
Nov 12 2015 19:15
unless there's some setting in PG that does single-lines-are-transactions
Matthew Draper
@matthewd
Nov 12 2015 19:15
.. in fact not even that, because that's deffered until you do a real operation :)
Jason Frey
@Fryguy
Nov 12 2015 19:15
implicit transactions or whatever DBs call it
Keenan Brock
@kbrock
Nov 12 2015 19:16
in other applications I've written - we didn't send a round trip begin end
Jason Frey
@Fryguy
Nov 12 2015 19:16
so, where you pay the penalty for BEGIN is the latency transfer
Keenan Brock
@kbrock
Nov 12 2015 19:16
heh. you could send "BEGIN ; INSTER ; COMMIT"
yes
I understand WHY it takes that time
Matthew Draper
@matthewd
Nov 12 2015 19:16
You don't technically have to send the transaction control commands for PG
Keenan Brock
@kbrock
Nov 12 2015 19:16
I understand WHY you need a "transaction" per say
I just think the tripple round trip is pretty expensive. and in C/Java, I never did it
Jason Frey
@Fryguy
Nov 12 2015 19:16
which is why we had such great gains with rails/rails#17937 ... in there you are doing literally BEGIN; <nothing;> COMMIT;
Matthew Draper
@matthewd
Nov 12 2015 19:17
But we (AR) do, because a save can incorporate more operations, and we can't know it in advance
Keenan Brock
@kbrock
Nov 12 2015 19:17
hmm
yes
kinda like the delete vs destroy
Matthew Draper
@matthewd
Nov 12 2015 19:17
@Fryguy does that mean we'd gain by just doing obj.save if obj.dirty? ?
Keenan Brock
@kbrock
Nov 12 2015 19:17
sounds like it
Jason Frey
@Fryguy
Nov 12 2015 19:18
if we coded that on our side yes, probably...especially now that dirty tracking is much better
Keenan Brock
@kbrock
Nov 12 2015 19:18
:monkey: patch
Jason Frey
@Fryguy
Nov 12 2015 19:18
the "fix" I did in Rails 2 was to delay the BEGIN until we sent the first real PG statement
Matthew Draper
@matthewd
Nov 12 2015 19:18
(again, because we (MIQ) know there aren't Other Things involved that might end up requiring work, in this particular case — not something we could do AR-wide)
Jason Frey
@Fryguy
Nov 12 2015 19:18
and if we got to the part where you would do a COMMIT, but had never issued the BEGIN, then we just ignore it
yes
my "fix" was AR wide
Keenan Brock
@kbrock
Nov 12 2015 19:19
yea
that looks like a good patch
even if you need to send the begin
you'd save at least 1 round trip
Jason Frey
@Fryguy
Nov 12 2015 19:19
(I also patched it there because I didn't want to tweak every place we did .save, .update_attributes, .update_attrbiute, etc)
technically, if you know what you're sending is the first PG statement, you can slam then together into a single call and remove 1 RT
on every statement
Matthew Draper
@matthewd
Nov 12 2015 19:20
Not if you're using parameters
Jason Frey
@Fryguy
Nov 12 2015 19:21
fair point
I'm still not sure parameterization is buying Rails anything
do you/they have metrics on that?
maybe particular apps benefit, but not others?
Matthew Draper
@matthewd
Nov 12 2015 19:21
String-mangling is expensive
Jason Frey
@Fryguy
Nov 12 2015 19:22
on the client side you mean? or in PG?
Matthew Draper
@matthewd
Nov 12 2015 19:22
Do you mean parameterization, or PREPARE?
Jason Frey
@Fryguy
Nov 12 2015 19:22
oh I guess I was thikning PREPARE
so, you can't multi-statement parameterized queries?
Matthew Draper
@matthewd
Nov 12 2015 19:23
Correct
But, I believe at a protocol level, there's no reason you need to wait for the server to ack your BEGIN before you send the next query
Jason Frey
@Fryguy
Nov 12 2015 19:24
interesting
Matthew Draper
@matthewd
Nov 12 2015 19:24
I'm just not so sure how easily you can convince libpq to run ahead like that
Jason Frey
@Fryguy
Nov 12 2015 19:24
even so, the gains I was touting in that rails issue weren't around that...they were more around not sending empty BEGIN/COMMIT pairs
Matthew Draper
@matthewd
Nov 12 2015 19:25
I had some students looking into it, but they ran into issues of some sort
Jason Frey
@Fryguy
Nov 12 2015 19:25

But, I believe at a protocol level, there's no reason you need to wait for the server to ack your BEGIN before you send the next query

That I have no idea

plus that transaction code likely is for all adapters, so you can't rely much on that anyway
I would think
Matthew Draper
@matthewd
Nov 12 2015 19:26
Specifically, I was looking into the possibility of wrapping all statements with individual savepoints (like psql can be told to), to get useful error-in-transaction semantics
Keenan Brock
@kbrock
Nov 12 2015 19:27
ooh - cool. so Host.first does not create a transaction, it is only Host.create
Matthew Draper
@matthewd
Nov 12 2015 19:28
(which did incidentally bring me to wanting to defer transaction start until it was actually needed)
@kbrock right… save has to because before/after callbacks need to live & die with the "main" UPDATE… most other stuff doesn't have that requirement
Including something like delete_all.. I would think.
Keenan Brock
@kbrock
Nov 12 2015 19:48
there is a lot of very cool / smart stuff in active record.
Matthew Draper
@matthewd
Nov 12 2015 19:48
There's a lot of other stuff too :trollface:
Keenan Brock
@kbrock
Nov 12 2015 19:49
heh
Keenan Brock
@kbrock
Nov 12 2015 20:41
@Fryguy C&U: can a provider have more than one of containers, infra and cloud? or it will only have 1 (e.g.: Metrics::Targets.capture_cloud_targets)
Jason Frey
@Fryguy
Nov 12 2015 20:55
according to my definition of providers, a provider has_many managers
where a manager can be anthing in the EMS table
I think the only ones that do that right now are OpenStack (cloud + optional infra) and Foreman (prov + config_mgmt)
Matthew Draper
@matthewd
Nov 12 2015 20:57
confirm
Jason Frey
@Fryguy
Nov 12 2015 20:57
eventually, I expect Amazon will have multiple regions under 1 prov umbrella
maybe VMware will get something like vcd for cloud and vsphere for infra
Keenan Brock
@kbrock
Nov 12 2015 20:58
ok, seems that calling into all 3 was adding an extra query and ~1-2% more objects. negligible, but was curious
Jason Frey
@Fryguy
Nov 12 2015 20:58
and as we blow out a networking manager, that will split out where appropriate (particularly on openstack where neutron will be it's own manager)
gotcha
that is interesting
Keenan Brock
@kbrock
Nov 12 2015 20:59
I'm specifically talking about capture_infra_targets + capture_cloud_targets + capture_container_targets
Jason Frey
@Fryguy
Nov 12 2015 20:59
oh interesting
I bet that should be made into something like manager.capture_targets
and then you would call that for each manager
Keenan Brock
@kbrock
Nov 12 2015 21:00
again - I'm still in the preliminaries
and it is tricky to actually know WHO is adding this extra stuff
Jason Frey
@Fryguy
Nov 12 2015 21:00
so Provider.capture_targets would be defined as managers.flat_map(&:capture_targets) or somethig like that
Keenan Brock
@kbrock
Nov 12 2015 21:00
the cpu used in that method is curious - but I could be measuring incorrectly
Jason Frey
@Fryguy
Nov 12 2015 21:00
yeah
Keenan Brock
@kbrock
Nov 12 2015 21:00
yea
cool
yea - that method is called capture_targets
Keenan Brock
@kbrock
Nov 12 2015 21:28
You know, perf_capture_enabled? seems expensive.
Also, I removed a few MiqPreloader.preload from Metrics::Targets and got the same performance. kind fun