These are chat archives for ManageIQ/manageiq/performance

5th
Feb 2016
Joe Rafaniello
@jrafanie
Feb 05 2016 16:11
@kbrock @dmetzger57 did you try migrating the customer database with master code?
Dennis Metzger
@dmetzger57
Feb 05 2016 16:12
I tried, but my UI workers refused to start after the import
Joe Rafaniello
@jrafanie
Feb 05 2016 16:12
I got stuck on the hosts_storages migration that Adam/Nick worked on and bailed on it for now
I had to comment out the migration :runner:
did the migrate work for you?
It failed because the user didn't own the table that we were trying to set the sequence on
I thought it might be a mac/dev setup thing... didn't want chase that yesterday
Keenan Brock
@kbrock
Feb 05 2016 16:13
@jrafanie I had odd errors on my import and a "cfme" role
the role rang a bell
it was something else
remember it from instructions 1-2 years ago
that was on my old database import
Joe Rafaniello
@jrafanie
Feb 05 2016 16:14
Yeah, the import had a few warnings because the user didn't exist
Keenan Brock
@kbrock
Feb 05 2016 16:14
wonder if that causes issue with upgrading
@jrafanie did you create the user? (I didn't)
Joe Rafaniello
@jrafanie
Feb 05 2016 16:14
the error I was concerned with was migrating on master code
No, I didn't
Keenan Brock
@kbrock
Feb 05 2016 16:14
heh
Dennis Metzger
@dmetzger57
Feb 05 2016 16:14
quite a few warnings
Joe Rafaniello
@jrafanie
Feb 05 2016 16:15
It failed on db/migrate/20150921204114_add_vmware_ro_datastores_to_hosts_storages.rb
Keenan Brock
@kbrock
Feb 05 2016 16:16
did you have a REGION file set to 509?
Joe Rafaniello
@jrafanie
Feb 05 2016 16:16
it also failed on the replication stuff because my server didn't have the replication host configured but that's ok
yes
and v2_key
Keenan Brock
@kbrock
Feb 05 2016 16:16
yea, replication is tricky
we had some issues before when the replication keys were v1_key, but that shouldn't be the case
you remember the error?
Dennis Metzger
@dmetzger57
Feb 05 2016 16:16
there was some indexing added recently because of the ro datastore wasn’t there, yes?
Keenan Brock
@kbrock
Feb 05 2016 16:17
I worked hard to make sure our stuff could upgrade from 5.0 to 5.5
Keenan Brock
@kbrock
Feb 05 2016 16:49
@jrafanie what is the problem with that migration?
Joe Rafaniello
@jrafanie
Feb 05 2016 16:51
done, now running the prior command to see if it's fixed
yay, I think I messed up yesterday
Keenan Brock
@kbrock
Feb 05 2016 16:51
well, it could be stale meta data
Keenan Brock
@kbrock
Feb 05 2016 17:07
upgrade = git checkout master ; rake db:migrate db:seed ?
maybe minging replication
Joe Rafaniello
@jrafanie
Feb 05 2016 17:10
yay, my fix_auth problem was user error
running the migrations again
Jason Frey
@Fryguy
Feb 05 2016 17:10
ok restoring now
Keenan Brock
@kbrock
Feb 05 2016 17:11
createdb -U root -T srcdb destdb #FTW
Jason Frey
@Fryguy
Feb 05 2016 17:12
I have a magic pg_r command locally that I wrote like 5 years ago, and I can't remember what's even in it haha
but it works, so I don't question it
Keenan Brock
@kbrock
Feb 05 2016 17:13
+1
Jason Frey
@Fryguy
Feb 05 2016 17:13
@jrafanie your backup is borked
Keenan Brock
@kbrock
Feb 05 2016 17:13
I started to write a backup / restore. hard to remember to backup with custom style
Jason Frey
@Fryguy
Feb 05 2016 17:13
it has your personal user as the owner of all the sequences
Keenan Brock
@kbrock
Feb 05 2016 17:13
nice
Jason Frey
@Fryguy
Feb 05 2016 17:14
might be ok though...I'll check it out after it finises restoring
ok its restored (only took 3 minutes..thanks for truncating @jrafanie!)
Joe Rafaniello
@jrafanie
Feb 05 2016 17:15

it has your personal user as the owner of all the sequences

Yes, that is why I think the migration failed for me, confirming

Keenan Brock
@kbrock
Feb 05 2016 17:15
ooh 337M vs 1.7G
Joe Rafaniello
@jrafanie
Feb 05 2016 17:15
yes
Keenan Brock
@kbrock
Feb 05 2016 17:16
think I'm going to look at perf capture on this bad baby
Jason Frey
@Fryguy
Feb 05 2016 17:16
you mean after we fix the performance? ;)
Joe Rafaniello
@jrafanie
Feb 05 2016 17:16
for testing performance stuff, 5 minute import vs. 30 minute is helpful ;-)
Keenan Brock
@kbrock
Feb 05 2016 17:16
yes
Jason Frey
@Fryguy
Feb 05 2016 17:16
I don't think the sequences are important...we are not adding new records
(well, not important for the speed testing...they are important for migration probably)
Has anyone tried performance testing the TreeBuilder outside of the UI?
Keenan Brock
@kbrock
Feb 05 2016 17:18
no
Jason Frey
@Fryguy
Feb 05 2016 17:18
k, when I did this in the past, that was really useful
Joe Rafaniello
@jrafanie
Feb 05 2016 17:20
@Fryguy I didn't because I was lazy, this is too easy: config.middleware.use Rack::RubyProf, :path => Rails.root.join('tmp/profile_master'), :printers => {RubyProf::CallStackPrinter => 'call_stack.html'}, :min_percent => 2
Jason Frey
@Fryguy
Feb 05 2016 17:20
yeah, the downside is that it blends two worlds together
which can also be an upside :D
Joe Rafaniello
@jrafanie
Feb 05 2016 17:20
yes, exactly
20% in tree builder might be only 3% in full UI
Jason Frey
@Fryguy
Feb 05 2016 17:21
yeah true
Jason Frey
@Fryguy
Feb 05 2016 18:04
finally got everything working locally...stupid eventmachine wouldn't compile on Ruby 2.0 :angry:
the fix was ln -s /usr/bin/gcc /usr/bin/gcc-4.2 (-‸ლ)
Daniel Berger
@djberg96
Feb 05 2016 18:06
I thought eventmachine was defunct
Jason Frey
@Fryguy
Feb 05 2016 18:07
maybe it is now, but not on old versions of ManageIQ
we're trying to fix a performance problem with CloudForms 5.4 (botvinnik, I think)
Joe Rafaniello
@jrafanie
Feb 05 2016 18:26
thin uses eventmachine
Jason Frey
@Fryguy
Feb 05 2016 18:26
what is the quick command to cherry-pick those 2 upstream PRs into the vmdb directory?
-Xsubtree or something, but I can't get it to work (must be something I'm doing)
Matthew Draper
@matthewd
Feb 05 2016 18:27
git cherry-pick -X subtree=vmdb/ SHA
Jason Frey
@Fryguy
Feb 05 2016 18:28
ah thank you
is that / required there?
Matthew Draper
@matthewd
Feb 05 2016 18:28
Probably not
Jason Frey
@Fryguy
Feb 05 2016 18:28
k
Joe Rafaniello
@jrafanie
Feb 05 2016 18:30
@matthewd is there a better place to enable Rack::RubyProf? ManageIQ/manageiq#6534 I don't want to put it in config/environments/* since you might want to run it in any environment
As @Fryguy is cleaning up the mess in application.rb, I don't want to pollute it
Keenan Brock
@kbrock
Feb 05 2016 18:31
@jrafanie I put it in initializers
Joe Rafaniello
@jrafanie
Feb 05 2016 18:32
do i have access to config there?
Keenan Brock
@kbrock
Feb 05 2016 18:32
require 'ruby-prof'
Rails.application.config.middleware.use Rack::RubyProf,
  :path => "#{ENV["HOME"]}/src/manageiq/vmdb/tmp/profile",
  :printers => {
Matthew Draper
@matthewd
Feb 05 2016 18:32
Yeah, I think an initializer is your other option, though application.rb is probably the most traditional choice
Joe Rafaniello
@jrafanie
Feb 05 2016 18:34
ok, do you peeps prefer initializer?
I don't want it to be so hidden that people can't find it
in case they want to use it
Jason Frey
@Fryguy
Feb 05 2016 18:34
either one is fine (to me)
Keenan Brock
@kbrock
Feb 05 2016 18:39
initializers allow me to git checkout and not have to stash
application.rb is more traditional / I first tried before having to git stash 10 times :smile:
aah Updating Type in EventStreams ==> this one will be fun
== 20150806211453 RenameEmsEventTableToEventStream: migrated (721.1085s) ======
Joe Rafaniello
@jrafanie
Feb 05 2016 18:44
Is that with the 10 GB ems_events table?
Keenan Brock
@kbrock
Feb 05 2016 18:44
yup
wow - we keep updating that table across many migrations
Joe Rafaniello
@jrafanie
Feb 05 2016 18:47
ok, you convinced me to use an initializer
Jason Frey
@Fryguy
Feb 05 2016 18:47
that's not really that bad considering how many events
:)
Keenan Brock
@kbrock
Feb 05 2016 18:59
Exception caught: Replication configuration missing
how do I clean out that config table?
Jason Frey
@Fryguy
Feb 05 2016 19:02
Configuration.delete_all?
Dennis Metzger
@dmetzger57
Feb 05 2016 19:42
Tested the two 5.4.z MRs (Infra to VM page rendering perf) using a local appliance with Alex’s Large vSphere env and on Jared’s appliance with the customer DB. Alex’s env is 3K VMs while the customer has ~13K.
In Jared’s env the page renders in 26.7 seconds after patching vs 44.9 seconds unpatched.
Using Alex’s env the page renders in 4.9 seconds after patching vs 11. seconds unpatched.
These times exclude the initial VM page rendering (which is also faster now, but still slower than subsequent renderings).
Keenan Brock
@kbrock
Feb 05 2016 19:47
@jrafanie did you put in a PR for icon performance bug?
@matthewd do we have a fix for:
ActionController::Base.helpers.image_path("16/#{icon}.png")
Matthew Draper
@matthewd
Feb 05 2016 19:50
I think it was "run in production, after an asset precompile"
Keenan Brock
@kbrock
Feb 05 2016 19:50
k
thnx
Matthew Draper
@matthewd
Feb 05 2016 19:51
As upstream, I'm intruiged, if image_path is really that slow in development.. but my impression is that it's not a real issue for us
Jason Frey
@Fryguy
Feb 05 2016 19:52
we found the reason value_to_integer is called 1.1 million times in the tree rendering
basically, it's this, on rails 3:
[12] pry(main)> e = ExtManagementSystem.first
  ExtManagementSystem Load (0.4ms)  SELECT "ext_management_systems".* FROM "ext_management_systems" LIMIT 1
  ExtManagementSystem Inst Including Associations (0.3ms - 1rows)
called value_to_integer
called value_to_integer
=> #<EmsVmware id: 509000000000002>
[13] pry(main)> e.id
called value_to_integer
=> 509000000000002
[14] pry(main)> e.id
called value_to_integer
=> 509000000000002
I patched AR in that version to puts "value_to_integer"right where it casts in that method
Joe Rafaniello
@jrafanie
Feb 05 2016 19:54
Yes, thanks @matthewd ... @kbrock I didn't run assets precompile in prod
Jason Frey
@Fryguy
Feb 05 2016 19:55
so, the casting of id to an integer isn't cached anywhere...it's called over and over...when subtree_arranged is walking the tree to insert the rows (a kind of insertion sort), it compares by the id and ends up calling Relationship#id over and over
Matthew Draper
@matthewd
Feb 05 2016 19:55
@Fryguy ping
Jason Frey
@Fryguy
Feb 05 2016 19:55
hi
on master, the id is called over and over again as well, but it doesn't cast, because of the way AttributeSet works...
oddly it's still 18% of the total time just calling .id over and over
Matthew Draper
@matthewd
Feb 05 2016 20:01
Is it being expensive because bignums?
Jason Frey
@Fryguy
Feb 05 2016 20:01
It's hard to tell
[15] pry(main)> 509000000000002.class
=> Fixnum
Joe Rafaniello
@jrafanie
Feb 05 2016 21:26
@matthewd changing draper's === to use is_a?(Draper::Decorator) instead of using respond_to? saves 1.2-1.5 seconds on master
before:
Completed 200 OK in 13827ms (Views: 2089.2ms | ActiveRecord: 2389.7ms)
Completed 200 OK in 10749ms (Views: 122.2ms | ActiveRecord: 2295.0ms)
Completed 200 OK in 10598ms (Views: 355.3ms | ActiveRecord: 2211.5ms)
Completed 200 OK in 10856ms (Views: 185.7ms | ActiveRecord: 2276.7ms)
Completed 200 OK in 10476ms (Views: 172.6ms | ActiveRecord: 2232.5ms)
Completed 200 OK in 10703ms (Views: 174.2ms | ActiveRecord: 2284.5ms)

after:
Completed 200 OK in 12016ms (Views: 1957.3ms | ActiveRecord: 2322.4ms)
Completed 200 OK in 8858ms (Views: 141.9ms | ActiveRecord: 2229.7ms)
Completed 200 OK in 9774ms (Views: 357.3ms | ActiveRecord: 2227.4ms)
Completed 200 OK in 9440ms (Views: 147.0ms | ActiveRecord: 2331.6ms)
Completed 200 OK in 9280ms (Views: 147.6ms | ActiveRecord: 2288.6ms)
Completed 200 OK in 9468ms (Views: 98.6ms | ActiveRecord: 2351.4ms)
Before:
      def ===(other)
        super || (other.respond_to?(:object) && super(other.object))
      end
After:
      def ===(other)
        super || (other.is_a?(Draper::Decorator) && super(other.object))
      end
Keenan Brock
@kbrock
Feb 05 2016 21:29
@jrafanie nice
@matthewd.is_a?(Draper) rather than @matthewd.responds_to?(:objectification)
Joe Rafaniello
@jrafanie
Feb 05 2016 21:33
lol
I wish someone would name a gem after me
but you are stale from 2009
Joe Rafaniello
@jrafanie
Feb 05 2016 21:34
https://rubygems.org/gems/joe
"Joe takes you from a gem specification to world domination in a single step."
Keenan Brock
@kbrock
Feb 05 2016 21:34
nice
Chris Arcand
@chrisarcand
Feb 05 2016 21:34
I’m a CLI tool to query Argentinean zip codes :trophy:
Keenan Brock
@kbrock
Feb 05 2016 21:35
hey - thats something
off to dig our our driveway (again)
Jason Frey
@Fryguy
Feb 05 2016 22:16

I’m a CLI tool to query Argentinean zip codes :trophy:

what? o_O

Daniel Berger
@djberg96
Feb 05 2016 22:37
They told him he could be whatever he wanted. He chose to be a CLI tool that queries Argentinian zip codes. You got a problem with that Fryguy?