These are chat archives for ManageIQ/manageiq/performance

25th
Nov 2015
Alex Krzos
@akrzos
Nov 25 2015 14:02
Perf update meeting - https://bluejeans.com/471610744/
Keenan Brock
@kbrock
Nov 25 2015 14:02
+1
Keenan Brock
@kbrock
Nov 25 2015 14:15
calls this 45 times on dashboard view - goes to db 45 times
user.rb:130
  def role_allows?(options = {})
    return false if miq_user_role.nil?
    feature = MiqProductFeature.find_by_identifier(options[:identifier])
Joe Rafaniello
@jrafanie
Nov 25 2015 14:20
Chris Arcand
@chrisarcand
Nov 25 2015 14:22
Such a fantastic gif.
Keenan Brock
@kbrock
Nov 25 2015 14:51
will product_features change? Can we totally cache them in memory?
Keenan Brock
@kbrock
Nov 25 2015 15:09
@jrafanie if we do pre-loading of proviers, would that reduce memory of workers?
Joe Rafaniello
@jrafanie
Nov 25 2015 15:10
preloading of what? our provider code?
Keenan Brock
@kbrock
Nov 25 2015 15:11
provider classes
maybe preload all the provider classes that are in the database?
Oleg Barenboim
@chessbyte
Nov 25 2015 15:11
how can pre-loading reduce memory??
Keenan Brock
@kbrock
Nov 25 2015 15:11
so each process doesn't need to load all the same classes
the classes would be shared?
Jason Frey
@Fryguy
Nov 25 2015 15:11
maybe once we fork workers
Joe Rafaniello
@jrafanie
Nov 25 2015 15:11
only if we do forking with COW... not following
Keenan Brock
@kbrock
Nov 25 2015 15:12
we're not COW?
no moo?
Jason Frey
@Fryguy
Nov 25 2015 15:12
:cow: :cow2:
Keenan Brock
@kbrock
Nov 25 2015 15:12
:)
Oleg Barenboim
@chessbyte
Nov 25 2015 15:12
@kbrock - never assume
Keenan Brock
@kbrock
Nov 25 2015 15:12
hmm, no butt emoji
I thought one of the main goals behind 2.2 was COW
Jason Frey
@Fryguy
Nov 25 2015 15:13
2.2?
Oleg Barenboim
@chessbyte
Nov 25 2015 15:13
@kbrock - full thoughts please
Keenan Brock
@kbrock
Nov 25 2015 15:13
sorry
I thought one of the pros of ruby 2.2.2/2.2.3 was better garbage collector so COW would work well
Jason Frey
@Fryguy
Nov 25 2015 15:14
COW only comes into play when you fork
Keenan Brock
@kbrock
Nov 25 2015 15:14
something about a bitmap
Jason Frey
@Fryguy
Nov 25 2015 15:14
our workers do not fork
at least not until LJ's PR comes in
Keenan Brock
@kbrock
Nov 25 2015 15:15
ok. guess I had thought that was in
there is still time in this release to get that in - right? ;)
Oleg Barenboim
@chessbyte
Nov 25 2015 15:15
no - ManageIQ/manageiq#3593
Keenan Brock
@kbrock
Nov 25 2015 15:15
thanks
Joe Rafaniello
@jrafanie
Nov 25 2015 15:15
2.0+ is COW friendly, but we need to fork first, ManageIQ/manageiq#3593, ON HOLD
Oleg Barenboim
@chessbyte
Nov 25 2015 15:17
@kbrock I thought it was too risky to slam that PR into this release, not knowing the performance implications
Keenan Brock
@kbrock
Nov 25 2015 15:17
sorry
yes
joke
don't merge
Jason Frey
@Fryguy
Nov 25 2015 15:18
we should merge for Darga sooner than later for burn-in time
Keenan Brock
@kbrock
Nov 25 2015 15:18
yes
Jason Frey
@Fryguy
Nov 25 2015 15:18
but not sure that PR is "ready" yet
Joe Rafaniello
@jrafanie
Nov 25 2015 15:18
yeah, i have a few todo items I think, haven't looked at it in a while
Oleg Barenboim
@chessbyte
Nov 25 2015 15:18
agree that we should merge it as soon as it is ready to allow for burn-in and performance measurements
Keenan Brock
@kbrock
Nov 25 2015 15:19
darn - I can't track down a framework. but someone wrote a worker framework that forks, receives messages from a queue, and handles safe rebooting children, closing db connections, but is minimal
we could learn from them
ok, actionable:
I'm seeing TONS of queries into features table
ideas how we can cut these down?
think our menubar is taking 45 queries
so on every page hit...
Oleg Barenboim
@chessbyte
Nov 25 2015 15:23
just cache the whole features table in memory - it may not last longer than 1 UI screen, but at least, you can get performance boost
Drew Bomhof
@syncrou
Nov 25 2015 15:26
@kbrock - I’ve often tried “safe rebooting” my children. It fixes nothing.
Keenan Brock
@kbrock
Nov 25 2015 15:26
yea
Keenan Brock
@kbrock
Nov 25 2015 15:34
@chessbyte cool. looks like these queries are to support 'hidden' features
Alex Krzos
@akrzos
Nov 25 2015 15:43
Joe Rafaniello
@jrafanie
Nov 25 2015 15:44
thanks @akrzos
akrzos @akrzos fired up his 100k database appliance too if anyone wants to check that out
Joe Rafaniello
@jrafanie
Nov 25 2015 19:45
I'm finding lots of wins in the memoist code base base on our use of it
FYI, we use memoist to cache the relationships between the CIs we care about
in lots of places, we need to flush the cache when changing the relationship context
and it's really expensive
Keenan Brock
@kbrock
Nov 25 2015 19:46
I challenge that statement
we WAY over flush caches
Joe Rafaniello
@jrafanie
Nov 25 2015 19:48
well, we I tried to eliminate the flush of the memoist cache but we need to flush it when we switch the context from "geneology" to "ems_metadata" and back to the former... so I eliminated one of the flush_cache calls but the one still needs to be there and it's really expensive
Keenan Brock
@kbrock
Nov 25 2015 19:48
yea -
Joe Rafaniello
@jrafanie
Nov 25 2015 19:48
especially when you have many memoized methods
Keenan Brock
@kbrock
Nov 25 2015 19:48
it is all the my_server.zone(true) stuff that I am reacting to
yes
Joe Rafaniello
@jrafanie
Nov 25 2015 19:48
So, that's not memoist, that's just caching
Keenan Brock
@kbrock
Nov 25 2015 19:48
+1
Joe Rafaniello
@jrafanie
Nov 25 2015 19:48
I think memoist is limitd to the relationship_mixin
I was seeing hundreds of thousands to a million objects created in some of those methods like: memoized_ivar_for, depending on the size of the report
Joe Rafaniello
@jrafanie
Nov 25 2015 21:19
with this memoist PR matthewrudy/memoist#36 and the ruport one ManageIQ/ruport#1, report generation is a bit faster (5%, and roughly 12% object allocations) but it depends on the report
one of the slower report "Folder to VMs Relationships" with @akrzos' db went from 44.78 to 42.47 seconds, ~29 million to ~25.4 million objects
Alex Krzos
@akrzos
Nov 25 2015 21:21
@jrafanie I believe that is one of the larger reports
let me dig through my benchmarks to see which ones were longest timing
Joe Rafaniello
@jrafanie
Nov 25 2015 21:22
There's still work to see if the caches in memoist are being invalidated more than needed but my initial work in that area didn't benefit us
Alex Krzos
@akrzos
Nov 25 2015 21:27
@jrafanie I have host summary for vms as the 2nd slowest
Folder to VMs Relationship as 3rd
1st place goes to User Accounts Windows, all of the simulated vms in my vcsims are "windows" vms
Joe Rafaniello
@jrafanie
Nov 25 2015 21:27
@Fryguy this is the initial "cache by relationship type" work I shelved for now since it didn't seem to help...we probably need to avoid busting the cache if nothing changed... https://github.com/ManageIQ/manageiq/compare/master...jrafanie:cache_by_relationship_type
not sure how to do that though
Alex Krzos
@akrzos
Nov 25 2015 21:28
Report:User Accounts - Windows 115.4458
Report:Host Summary for VMs 94.3462
Report:Folder to VMs Relationships 83.026
Joe Rafaniello
@jrafanie
Nov 25 2015 21:28
:trophy: Congratulations "User Accounts Windows"
Alex Krzos
@akrzos
Nov 25 2015 21:28
on vmware xlarge benchmarked with 5.5.0.11
let me pull the memory usages for those as well
Joe Rafaniello
@jrafanie
Nov 25 2015 21:29
ok
I"m thinking my changes on ruport/memoist would have more impact on memory than time
object allocations/gc don't take up that much time on ruby 2.2
Alex Krzos
@akrzos
Nov 25 2015 21:30
Report:User Accounts - Windows 723.2773438
Report:Host Summary for VMs 658.8984375
Report:Folder to VMs Relationships 534.2304688
Maximum in MiB ^
User Accounts still #1 for rss memory
Joe Rafaniello
@jrafanie
Nov 25 2015 21:30
I'd be curious if those PR make a dent
Alex Krzos
@akrzos
Nov 25 2015 21:30
Host Summary still #2
Joe Rafaniello
@jrafanie
Nov 25 2015 21:30
well, we'll try them out next week ;-)
Alex Krzos
@akrzos
Nov 25 2015 21:31
Folders moved to 8th place for rss memory usage
Maybe we pull in a master appliance and run just my reports benchmark against the pr
Joe Rafaniello
@jrafanie
Nov 25 2015 21:32
it's easy on master with git/bundler
diff --git a/Gemfile b/Gemfile
index fec7240..b433a51 100644
--- a/Gemfile
+++ b/Gemfile
@@ -24,7 +24,7 @@ gem "css_splitter"
 gem "sass-rails"

 # Vendored and required
-gem "ruport",                         "=1.7.0",                       :git => "git://github.com/ManageIQ/ruport.git", :tag => "v1.7.0-2"
+gem "ruport",                         :github => "jrafanie/ruport", :branch => "faster_data_record_new"

 # HACK: Force color to be required before azure-armrest. color is lazy required
 #   by ruport.  However, due to a bug in color, it detects the top level
diff --git a/gems/pending/Gemfile b/gems/pending/Gemfile
index 6150ea7..74740d1 100644
--- a/gems/pending/Gemfile
+++ b/gems/pending/Gemfile
@@ -27,7 +27,7 @@ gem "kubeclient",              "=0.8.0",            :require => false
 gem "hawkular-client",         "~>0.1.2",           :require => false
 gem "linux_admin",             "~>0.12.1",          :require => false
 gem "log4r",                   "=1.1.8",            :require => false
-gem "memoist",                 "~>0.11.0",          :require => false
+gem "memoist",                           :require => false, :github => "jrafanie/memoist", :branch => "faster"
 gem "memory_buffer",           ">=0.1.0",           :require => false
 gem "more_core_extensions",    "~>1.2.0",           :require => false
 gem "net-sftp",                "~>2.1.2",           :require => false
@akrzos ping me if you need any help...
I gotta run, happy turkey day everyone!
Alex Krzos
@akrzos
Nov 25 2015 21:33
I kinda recall editing the Gemfile leading to a whole lot of pain last time
Keenan Brock
@kbrock
Nov 25 2015 21:33
have a good one
Joe Rafaniello
@jrafanie
Nov 25 2015 21:34
yes, don't edit the Gemfile downstream
only upstream
Alex Krzos
@akrzos
Nov 25 2015 21:34
Yeah it might have been a downstream appliance
Jason Frey
@Fryguy
Nov 25 2015 22:39
If be shocked if those features queries aren't already cached with the query cache on the UI side. Caching yourself should not be a win, I would think
Keenan Brock
@kbrock
Nov 25 2015 22:39
+1
Jason Frey
@Fryguy
Nov 25 2015 22:39
If you see CACHE in dev log, then you won't gain anything
Keenan Brock
@kbrock
Nov 25 2015 22:40
@Fryguy yes, it is already cached - I'm not "adding anything"
just using what is already there
Jason Frey
@Fryguy
Nov 25 2015 22:40
Not saying we shouldn't try to eliminate the duplicate queries though
Yeah
But it's much less priority I would think
Keenan Brock
@kbrock
Nov 25 2015 22:41
I just got rid of 40 queries per page
every page
Jason Frey
@Fryguy
Nov 25 2015 22:43
@jrafanie that is great... Exactly what I was thinking... Better in fact, because you still leverage memoist and swap out the values
Any estimates on savings?
Keenan Brock
@kbrock
Nov 25 2015 22:43
300-400ms / page
Jason Frey
@Fryguy
Nov 25 2015 22:44
@kbrock Nice!
OK :meat_on_bone: time with the family... Happy Thanksgiving everyone!
Joe Rafaniello
@jrafanie
Nov 25 2015 22:45
@Fryguy re: cache by rel type, savings were not good
Jason Frey
@Fryguy
Nov 25 2015 22:46
@jrafanie.... Interesting... Probably because the bulk code avoid it
Joe Rafaniello
@jrafanie
Nov 25 2015 22:46
I need to prevent flush cache to make it with it
Worth it
Jason Frey
@Fryguy
Nov 25 2015 22:47
I wonder if you revert the bulk code if you'd see better numbers (better than pre bulk, I mean)
@jrafanie if may also be savings where we actually change type... I'm not sure refresh changes type all that much
But genealogy does
Joe Rafaniello
@jrafanie
Nov 25 2015 22:48
I got more savings by making the flush and prime less expensive in memoist. See pr on memoist for details
On phone, have a happy Thanksgiving!
Jason Frey
@Fryguy
Nov 25 2015 22:51
Nice! Just read the PR... Good work!
Joe Rafaniello
@jrafanie
Nov 25 2015 22:54
Thanks, need more measurements to make. more changes
Jason Frey
@Fryguy
Nov 25 2015 23:00
Diminishing returns though, probably
Keenan Brock
@kbrock
Nov 25 2015 23:03
@Fryguy nope - looks like the numbers I was quoting from development, in production it only saves 60ms: 328ms -> 285ms (15%) - but on every page /cc @jrafanie
also found 3 bugs while I was there though :)