These are chat archives for ManageIQ/manageiq/performance

27th
May 2016
Keenan Brock
@kbrock
May 27 2016 13:14
@chrisarcand you have a sec for performance?
Chris Arcand
@chrisarcand
May 27 2016 13:14
shoot
Keenan Brock
@kbrock
May 27 2016 13:15
on every page, we lookup the user
and their group, tenant, role
a user currently set's their current group (I want to get rid of this so badly... that is why I'm excite about your entitlements change and not having a current user... but anyway)
how can I do a single query (very wide) that will assign a user's group, role, and (maybe tenant)?
and assign it into the rails model so rails is happy
Chris Arcand
@chrisarcand
May 27 2016 13:17
I’m not sure if that’s possible or wise
Keenan Brock
@kbrock
May 27 2016 13:18
With includes you used to be able to assign it
ok, why not wise?
Chris Arcand
@chrisarcand
May 27 2016 13:18
Seems pretty overeaching.
Keenan Brock
@kbrock
May 27 2016 13:18
when a user comes into the sight
Chris Arcand
@chrisarcand
May 27 2016 13:18
I haven’t played with them a ton, but one thought is an sql view to encompass the ‘current’ aspect of users stuff
Keenan Brock
@kbrock
May 27 2016 13:19
we always care what their group, tenant, and role is
but that would be a single object
Chris Arcand
@chrisarcand
May 27 2016 13:19
yeah?
what does assignment of this stuff have to do with the single query lookup, btw? :confused:
Keenan Brock
@kbrock
May 27 2016 13:20
I was hoping to reduce the base number of queries for every page
Chris Arcand
@chrisarcand
May 27 2016 13:20
Oh you mean just aggregation of the model we use there, not actually assigning and saving new current columns
Yeah seems legit
I wonder if there’s some way to make a big left join (like you describe) and still split out AR objects from that. I assume you’ll still need the separate group for whatever session[:group] is
Jason Frey
@Fryguy
May 27 2016 13:44
What is the minimum queries you've gotten it to so far? Sounds like 3 or 4, maybe?
I ask because it worth trying to get it lower, or are there biggest bottlenecks than SQL queries that point?
s/
Keenan Brock
@kbrock
May 27 2016 14:02
good point jason
every key press = 400ms
down to 22 queries,
100ms = looking up "who am I"
rbac counts = 50ms? (hard to tell)
that 100ms tax seemed high, so was thinking we could reduce and it would affect every page
but maybe only minor wins
Jason Frey
@Fryguy
May 27 2016 14:06
Why do keypress inputs need all that Rbac...I would think session I'd is plenty
Or even a user for that matter
Keenan Brock
@kbrock
May 27 2016 14:06
yea, lookin at reducing the rbacs - they are down to 25ms each
or if it has rows, then 100ms
Jason Frey
@Fryguy
May 27 2016 14:07
I'm not talking reducing, I'm talking removing entirely
Keenan Brock
@kbrock
May 27 2016 14:07
reducing the number of rbacs, removing the unneeded one - tomato/potato
wait - are you talking authentication or rbacs?
Jason Frey
@Fryguy
May 27 2016 14:07
Both
Keenan Brock
@kbrock
May 27 2016 14:08
we like to know who the user is. that is ~100ms hmm. (maybe 75ms?)
Jason Frey
@Fryguy
May 27 2016 14:08
Like if I'm entering a form. The only thing changing is the text box I'm typing in
And the round trip to the server us for updating the session
Keenan Brock
@kbrock
May 27 2016 14:08
it could interact with other entries on the form. so it is tricky to make that generic, powerful, and optimal
Jason Frey
@Fryguy
May 27 2016 14:09
So all it needs of a preauthenticated session and that's it
Hmmmm
I guess then only load those values on demand?
Keenan Brock
@kbrock
May 27 2016 14:09
load on demand vs preload/up front - you know that is a difficult optimization :)
@Fryguy am I expecting a select from settings_changes for every hit to the server?
Greg McCullough
@gmcculloug
May 27 2016 14:11
@kbrock is this the provisioning dialogs still or something else?
Keenan Brock
@kbrock
May 27 2016 14:11
@gmcculloug provisioning dialogs
Greg McCullough
@gmcculloug
May 27 2016 14:11
I do not see where typing in the edit box ever needs to interact with another field.
drop-down select, list-view select, yes. but edit fields no. unless we are just talking about updating the session object
Keenan Brock
@kbrock
May 27 2016 14:13
process_filter seems to be calling rbac for every value even blank ones
I did reduce it a bunch, but... yea
Jason Frey
@Fryguy
May 27 2016 14:20
@kbrock re settings changes, yes but only in dev mode
In prod I would not expect that
Greg McCullough
@gmcculloug
May 27 2016 14:21
the only text field I can thinking of that interacts with other field while typing is the “Vm Description” field on the Catalog tab. it shows how many characters you have typed out of the 100 character limit. i.e. "VM Description (4 / 100)"
Jason Frey
@Fryguy
May 27 2016 14:23
Question regarding this... When a key is pressed in an edit box, what gets sent to the server... The whole form, or just the text, or something else?
And what is the expected response from the server
Keenan Brock
@kbrock
May 27 2016 14:34
lol - jason. I have the same question.
Lucy Fu
@lfu
May 27 2016 14:56
What is wrong with this code? it did not stop.
22 update_count = nil
23 done = 0
24 until update_count == 0
S> 25 update_count = EventStream.limit(500_000).update_all(:type => 'EmsEvent')
26 done += update_count
S> 27 say "#{done.to_s.reverse.gsub(/...(?=.)/,'\&,').reverse} rows", :subitem if update_count > 0
28 end
it went to:
-> 184,000,000 rows
-> 184,500,000 rows
Jason Frey
@Fryguy
May 27 2016 14:56
you are not batching the rows properly
it's taking the same 500_000 over and over
try this instead:
Lucy Fu
@lfu
May 27 2016 14:57
oh, i missis where(:type=>nil)
Jason Frey
@Fryguy
May 27 2016 14:57
oh yeah...that'll do it :)
Keenan Brock
@kbrock
May 27 2016 15:18
@Fryguy ok, so with travis builds, there is a after script that saves files to the cache.
I wonder with our different builds, since they use different gemfiles, do those files change and cause it to always update the gem cache?
hmm. looks like gemfiles being different is ok
but git is dinging us 30-50 seconds very often. Can we move the git links to a gem?
and/or can we peg the git links to an actual tag?
Keenan Brock
@kbrock
May 27 2016 21:18
@jrafanie @gtanzillo Remember the Rbac find statement that we dropped? ==> ManageIQ/manageiq#8718
Well, turns out that most of ActsAsActiveRecord do not implement find
Joe Rafaniello
@jrafanie
May 27 2016 21:18
Not really :worried:
hehe
Keenan Brock
@kbrock
May 27 2016 21:18
model defines works comments
chargeback.rb N
cim_base_storage_extent.rb scope Y
live_metric.rb find Y
pglogical_subscription.rb find Y can be converted to sql
vim_performance_planning.rb N
vim_performance_trend.rb N
@jrafanie you remember saying well, if they don't implement find, then rbac won't do anything for tenants or limiting at all?
-    if klass.respond_to?(:find)
-      targets, total_count, auth_count = find_targets_with_rbac(klass, scope, user_filters, find_options, user, miq_group)
-    else
-      total_count = targets.length
-      auth_count  = targets.length
-    end
+    targets, total_count, auth_count = find_targets_with_rbac(klass, scope, user_filters, find_options, user, miq_group)
Joe Rafaniello
@jrafanie
May 27 2016 21:19
I need to look at my github contributions to remember what i did yesterday ;-)
Jason Frey
@Fryguy
May 27 2016 21:20
pglogical can't be covered to AR... We tried
Keenan Brock
@kbrock
May 27 2016 21:20
I got 50% of the way
it can
but not my priority
it is working
it is also VERY N+1
anyway
Jason Frey
@Fryguy
May 27 2016 21:20
It doesn't use a table... It's a function
Keenan Brock
@kbrock
May 27 2016 21:20
chargeback, vim performance planning and trend are AAARM but don't implement find
Jason Frey
@Fryguy
May 27 2016 21:20
And AR can't support functions
Keenan Brock
@kbrock
May 27 2016 21:21
yea, I know... I had it working
it was a little botched
but wasn't n+1
anyway
sorry
Jason Frey
@Fryguy
May 27 2016 21:21
Interesting...cc @carbonin
Keenan Brock
@kbrock
May 27 2016 21:21
@Fryguy can you verify - AAARM should implement find?
or all or SOMETHING that represents "look this up in the 'table'"
Jason Frey
@Fryguy
May 27 2016 21:27
Lol I was in a tunnel...
So, no it doesn't require find
Some of those classes just have .build_something_something methods
Jason Frey
@Fryguy
May 27 2016 21:33
The purpose is to have instances that look like AR models and respond to the expected methods, and have typed "columns"
Keenan Brock
@kbrock
May 27 2016 21:33
but how to "fetch" data is up for grabs
Jason Frey
@Fryguy
May 27 2016 21:33
So how they are created can be any way the eventually calls .new
Keenan Brock
@kbrock
May 27 2016 21:34
ok
so why would we call rbac on these then?
Jason Frey
@Fryguy
May 27 2016 21:34
I'm not sure
Maybe just for user context
Keenan Brock
@kbrock
May 27 2016 21:35
rbac won't do anything without find
well, it will apply a filter
and get a count
ok, thanks
Keenan Brock
@kbrock
May 27 2016 22:22

Yay @yrudman found the line ==> here which I find very confusing.
We run rbac and generate non rbac classes. Then we run rbac on THAT. Seems pagination (and maybe filter) is the only thing the rbac routine could possibly do.

@Fryguy @jrafanie thoughts on converting that to something other than rbac? (we already know it will skip the find and the core of rbac)

hmm, this seems like a non-darga change. I'll set that one up for E and roll back this change
Keenan Brock
@kbrock
May 27 2016 22:39
@gmcullough took another 2 queries out of the requests: ManageIQ/manageiq#9017