These are chat archives for ManageIQ/manageiq/performance

1st
Oct 2015
Keenan Brock
@kbrock
Oct 01 2015 12:29
@jrafanie ping if you can spare any cycles, need a little help getting up to speed capturing memory around running a refresh
Dennis Metzger
@dmetzger57
Oct 01 2015 12:31
I'd be interested in watching the proceedings
Keenan Brock
@kbrock
Oct 01 2015 12:34
@akrzos you in this early in the day?
Alex Krzos
@akrzos
Oct 01 2015 13:29
@kbrock yes usually
however I forgot to open gitter
Keenan Brock
@kbrock
Oct 01 2015 13:32
for refresh performance, can I point to one of your small "servers"?
Alex Krzos
@akrzos
Oct 01 2015 13:32
@kbrock sure
Joe Rafaniello
@jrafanie
Oct 01 2015 13:49
@kbrock sure, what type of info do you want to get? resident/virtual memory before/after?
Joe Rafaniello
@jrafanie
Oct 01 2015 14:22
@akrzos Can you adjust your script to keep track of MiqProcess.processInfo[:memory_size] and MiqProcess.processInfo[:memory_usage], not just the latter
Also, do you have a baseline comparison of an idle refresh worker 5.4 to 5.5?
Alex Krzos
@akrzos
Oct 01 2015 14:26
@jrafanie I can add that to track as well, :memory_size is virtual memory right?
Joe Rafaniello
@jrafanie
Oct 01 2015 14:26
Yes
thanks, I think that would give us a full picture, in addition to the baseline of an idle worker
Alex Krzos
@akrzos
Oct 01 2015 14:26
I can get those numbers of an idle refresh worker 5.4/5.5 but I don't have an automated method around that, if there is value in that kinda metric, I can create some automation around it
For the idle worker, it should be a newly spawned worker that had not completed any refresh jobs yet right?
Joe Rafaniello
@jrafanie
Oct 01 2015 14:27
@akrzos I would almost assume that we’d want ongoing automated benchmarks of an idle instantiated worker
even a basic generic worker woudl be helpful
As soon as we add code that loads lots of stuff, all workers woudl probably see it
Alex Krzos
@akrzos
Oct 01 2015 14:31
gotcha, so idle generic workers and priority workers should be easy
Joe Rafaniello
@jrafanie
Oct 01 2015 14:31
@akrzos note, I have an old pull request to take advantage of copy on write by using fork to create workers, so we can possibly load up lots of our code, then fork and many of the workers would share some of the initial code in a single place in emory
Alex Krzos
@akrzos
Oct 01 2015 14:32
@jrafanie my understanding is that only Ruby 2.0 appliances and above should benefit from COW
so for refresh worker, it is only spawned if the correct role is enabled and a provider is added, is there hooks that I can have an appliance create those workers without a provider and without any workload being queued
Joe Rafaniello
@jrafanie
Oct 01 2015 14:34
yes
yes, was an answer to the 2.0 question...
Alex Krzos
@akrzos
Oct 01 2015 14:34
gotcha
Joe Rafaniello
@jrafanie
Oct 01 2015 14:36
note, COW with fork makes diagnostics hard… top and most tools report shared memory as memory for the process so the only useful measure is the before/after total used memory
Don’t think there’s an easy way to make refresh workers start without an ems
Alex Krzos
@akrzos
Oct 01 2015 14:38
@jrafanie I believe we can measure RSS + shared memory per worker, though then total memory for all workers becomes hard to measure, since we aren't sure what is shared amongst each worker process plus evmserverd
Matthew Draper
@matthewd
Oct 01 2015 14:43
Yeah, I think there's a thing you can do to identify the shared memory, then share/divide the blame across the participating processes
.. but does require more cleverness than top can bring to the table
Alex Krzos
@akrzos
Oct 01 2015 14:55
I'm not sure /proc/pid/stat exposes the shared memory though which is what MiqProcess is reading
Joe Rafaniello
@jrafanie
Oct 01 2015 15:13
yeah, @akrzos, I couldn’t find anything that measured per process memory where any shared memory is only counted once
Alex Krzos
@akrzos
Oct 01 2015 15:18
@jrafanie Setting up for objspace measurements now
@jrafanie What specific measurements before/after should I gather? I'm running this on 5.4 and 5.5
Looks like tracing object allocations should be most useful
After that I'll setup a manual test for idle refresh worker memory size between 5.4 and 5.5 though
Alex Krzos
@akrzos
Oct 01 2015 15:35
@jrafanie I don't think 5.4 can trace object allocations:
Joe Rafaniello
@jrafanie
Oct 01 2015 15:35
@akrzos careful, not sure if all of the objspace info is in the 5.4
ObjectSpace.count_tdata_objects, ObjectSpace.count_objects_size, ObjectSpace.memsize_of_all(String), ObjectSpace.memsize_of_all(String) might all be useful
@matthewd is there a one that gives you counts by cruby type?
ie, ObjectSpace.count_objects_size but for count, not size
Joe Rafaniello
@jrafanie
Oct 01 2015 15:39
@akrzos I don’t know how much overhead allocation start/stop has… but the allocation_sourcefile(obj), allocation_sourceline(obj), and memsize_of(obj) could be useful once we find objects we want to inspect
Matthew Draper
@matthewd
Oct 01 2015 15:43
@jrafanie not that I immediately know of… though you can just each_object it ¯\_(ツ)_/¯
Alex Krzos
@akrzos
Oct 01 2015 15:43
5.4 objectspace https://gist.github.com/akrzos/a4ea09054b98ca03dbce no tracing, let me do the same for 5.5
Joe Rafaniello
@jrafanie
Oct 01 2015 15:46
Doh, sorry @akrzos ObjectSpace.memsize_of_all(String) and ObjectSpace.memsize_of_all
Alex Krzos
@akrzos
Oct 01 2015 15:47
@jrafanie no worries I can set this back up fairly quickly
Joe Rafaniello
@jrafanie
Oct 01 2015 15:47
plus, adding the virt/res memory bookends would be helpful
MiqProcess.processInfo[:memory_size] and MiqProcess.processInfo[:memory_usage]
Alex Krzos
@akrzos
Oct 01 2015 15:48
Do you want GC.stat as well?
Joe Rafaniello
@jrafanie
Oct 01 2015 15:50
sure
Alex Krzos
@akrzos
Oct 01 2015 15:53
Alright running:
ActiveRecord::Base.logger.level = 1
require "objspace"
mrss_start = MiqProcess.processInfo()[:memory_usage]
vrss_start = MiqProcess.processInfo[:memory_size]
GC.stat
e = ExtManagementSystem.find_by_name('vmware-small')
timing = Benchmark.realtime { EmsRefresh.refresh e }
mrss_end = MiqProcess.processInfo()[:memory_usage]
vrss_end = MiqProcess.processInfo[:memory_size]
GC.stat
mrss_change = mrss_end - mrss_start
vrss_change = vrss_end - vrss_start
puts "#{mrss_start}, #{mrss_end}, #{mrss_change}"
puts "#{vrss_start}, #{vrss_end}, #{vrss_change}"
puts timing
ObjectSpace.count_tdata_objects
ObjectSpace.count_objects_size
ObjectSpace.memsize_of_all(String)
ObjectSpace.memsize_of_all
Thats the small environment initial refresh
5.5 twice as much virtual memory allocated, Objspace.memsize_of_all is larger by around 56MiB
on 5.5
Alex Krzos
@akrzos
Oct 01 2015 16:07
1.5 million t_object on 5.4 vs 8.7million on 5.5
Jason Frey
@Fryguy
Oct 01 2015 16:07
interesting
there's also double the number of classes
it may just be more eager loading of some sort...perhaps the sti loader or something
Matthew Draper
@matthewd
Oct 01 2015 16:11
Yeah, I think the STI loader is a little bit over-eager at the moment :pensive:
Joe Rafaniello
@jrafanie
Oct 01 2015 16:11
ruby_parser creates lots of objects too
Jason Frey
@Fryguy
Oct 01 2015 16:11
{
:TOTAL      => [106144074, 171627703],
:T_ARRAY    => [1656064,   14708368],
:T_BIGNUM   => [nil,       38400],
:T_CLASS    => [6021880,   12017192],
:T_COMPLEX  => [nil,       40],
:T_DATA     => [70442699,  72754121],
:T_FILE     => [13376,     66464],
:T_FLOAT    => [nil,       440],
:T_HASH     => [11525944,  25763296],
:T_ICLASS   => [nil,       187920],
:T_MATCH    => [79920,     722624],
:T_MODULE   => [1920216,   2902400],
:T_NODE     => [nil,       3183672],
:T_OBJECT   => [1538040,   8776496],
:T_RATIONAL => [nil,       44960],
:T_REGEXP   => [2298519,   3227319],
:T_STRING   => [10637336,  26678503],
:T_STRUCT   => [10080,     443168],
:T_SYMBOL   => [nil,       112320]
}
that's the comparison breakdown
Oleg Barenboim
@chessbyte
Oct 01 2015 16:11
probably all the VMware WSDL stuff is getting loaded -- which is huge
Matthew Draper
@matthewd
Oct 01 2015 16:12
That's… a lot of arrays.
Jason Frey
@Fryguy
Oct 01 2015 16:14
@chessbyte that particular thing should be the same because the test is against VMware
Alex Krzos
@akrzos
Oct 01 2015 16:14
I can run through with rhevm as well for comparsion
Joe Rafaniello
@jrafanie
Oct 01 2015 16:15
want to try allocation_tracer on ruby 2.2 @akrzos ?
it frequently segfaults but when it works, it gives us more information ;-)
Alex Krzos
@akrzos
Oct 01 2015 16:16
I grabbed it once, just unsorted for 5.5 - https://gist.github.com/akrzos/5e28f3474163ab3d9fcb
Joe Rafaniello
@jrafanie
Oct 01 2015 16:19
to see top allocation locations, you can do something like this:
def trace
  require 'allocation_tracer'

  ObjectSpace::AllocationTracer.setup(%i{path line type class})
  result = ObjectSpace::AllocationTracer.trace do
    yield
    nil
  end

  result.each {|k, v|
    puts "#{v[0]}\t#{k[0]}:#{k[1]}\t#{k[2]}\t#{k[3]}\t#{k[4]}"
  }
end


e = ExtManagementSystem.find_by_name('vmware-small')

trace { EmsRefresh.refresh e }
and run that via: bundle exec rails r this_script | sort -nr
you’d need allocation_tracer in your bundle via Gemfile.dev.rb or adding it to the end of the Gemfile
Alex Krzos
@akrzos
Oct 01 2015 16:22
@jrafanie I'll give that a shot
Joe Rafaniello
@jrafanie
Oct 01 2015 16:24
Just loading our rails environment on mac shows this:
348583    /Users/joerafaniello/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274    T_NODE
228347    /Users/joerafaniello/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274    T_STRING    String
76003    /Users/joerafaniello/.rubies/ruby-2.2.3/lib/ruby/2.2.0/racc/parser.rb:258    T_ARRAY    Array
65575    poparser.ry:148    T_STRING    String
56860    poparser.ry:162    T_STRING    String
56859    poparser.ry:163    T_STRING    String
54538    poparser.ry:139    T_STRING    String
42229    /Users/joerafaniello/.gem/ruby/2.2.3/gems/json-1.8.3/lib/json/common.rb:155    T_STRING    String
29824    /Users/joerafaniello/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/journey/nodes/node.rb:33    T_STRING    String
28607    poparser.ry:189    T_STRING    String
28606    poparser.ry:191    T_STRING    String
28606    poparser.ry:190    T_STRING    String
28606    poparser.ry:190    T_ARRAY    Array
28606    poparser.ry:189    T_MATCH    MatchData
28606    poparser.ry:148    T_NODE
27851    poparser.ry:142    T_STRING    String
27433    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby22_parser.rb:1851    T_STRING    String
27373    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby22_parser.rb:903    T_STRING    String
27197    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby21_parser.rb:1833    T_STRING    String
27166    poparser.ry:141    T_STRING    String
27166    poparser.ry:140    T_STRING    String
27165    poparser.ry:139    T_NODE
27137    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby21_parser.rb:894    T_STRING    String
26508    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby20_parser.rb:1791    T_STRING    String
26476    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby20_parser.rb:872    T_STRING    String
25920    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby18_parser.rb:1688    T_STRING    String
25845    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby18_parser.rb:814    T_STRING    String
25616    /Users/joerafaniello/.gem/ruby/2.2.3/gems/psych-2.0.15/lib/psych.rb:376    T_STRING    String
24519    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby19_parser.rb:1659    T_STRING    String
24453    /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby19_parser.rb:804    T_STRING    String
23374    /Users/joerafaniello/.gem/ruby/2.2.3/gems/psych-2.0.15/lib/psych/tree_builder.rb:75    T_OBJECT    Psych::Nodes::Scalar
19827    /Users/joerafaniello/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274    T_ARRAY
19655    /Users/joerafaniello/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/journey/gtg/builder.rb:97    T_ARRAY    Array
18591    /Users/joerafaniello/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274    T_ARRAY    Array
17897    /Users/joerafaniello/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/journey/gtg/builder.rb:140    T_ARRAY    Array
17895    /Users/joerafaniello/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/journey/gtg/builder.rb:116    T_ARRAY    Array
15308    /Users/joerafaniello/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/journey/parser.rb:137    T_OBJECT    ActionDispatch::Journey::Nodes::Cat
14318    /Users/joerafaniello/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/journey/gtg/builder.rb:135    T_ARRAY    Array
We’d have to navigate ObjectSpace.each_object to determine the size of each of those objects
Pizza time, be back shortly
Alex Krzos
@akrzos
Oct 01 2015 16:45
Alright so few issues trying to get allocation_tracer running on 5.5.0.1 appliance - Could not find gem 'allocation_tracer (>= 0) ruby' in any of the gem sources listed in your Gemfile or available on this machine.
Jason Frey
@Fryguy
Oct 01 2015 16:55
I wish the gettext code would only load translations when it needs them
I think it's loading all of them
Dan Clarizio
@dclarizio
Oct 01 2015 16:55
@Fryguy as in ALL languages too???
Jason Frey
@Fryguy
Oct 01 2015 16:57
grosser/fast_gettext#75
Jason Frey
@Fryguy
Oct 01 2015 17:03
@jrafanie, Can you change your tester to see what delay loading might buy us? You can create a config/supported_locales.yml file and just put the following in it:
---
- en
Actually, nevermind...it's too late by the time we set that
Joe Rafaniello
@jrafanie
Oct 01 2015 17:07
@akrzos oh doh, you can’t add it to the gemfile on 5.5 because of the bundler issue @Fryguy and I haven’t fixed yet
you’d have to manually modify the Gemfile.lock to add it in two places
```
143     akami (1.2.2)
144       gyoku (>= 0.4.0)
145       nokogiri
146     allocation_tracer (0.6.3)
then also in the dependencies section
647 DEPENDENCIES
648   activerecord (~> 4.2.3)
649   activerecord-deprecated_finders (~> 1.0.4)
650   activesupport (~> 4.2.3)
651   acts_as_list (~> 0.1.4)
652   acts_as_tree (~> 2.1.0)
653   allocation_tracer
654   american_date
and hopefully you can use it in the bundle
we could use the upstream appliance to do that
Oleg Barenboim
@chessbyte
Oct 01 2015 17:36
@tenderlove cracks me up
AdequateRecord and now the "Not Very Good MRI Heap Analyzer"
Keenan Brock
@kbrock
Oct 01 2015 18:03
:)
Alex Krzos
@akrzos
Oct 01 2015 18:24
@jrafanie Does order matter in the Gmefile.lock? My line 143 looks different
Keenan Brock
@kbrock
Oct 01 2015 18:25
@akrzos no
Joe Rafaniello
@jrafanie
Oct 01 2015 18:30
no, i just gave context for reference
Keenan Brock
@kbrock
Oct 01 2015 19:34
5.4.z:
memsize: 11451895, , count: 59927
memsize: 10163487, /Users/kbrock/.gem/ruby/2.2.3/bundler/gems/rails-4842a8377644/activesupport/lib/active_support/dependencies.rb, count: 22093
memsize: 5336738, /opt/rubies/ruby-2.2.3/lib/ruby/2.2.0/rubygems/specification.rb, count: 83143
memsize: 1856408, /Users/kbrock/.gem/ruby/2.2.3/bundler/gems/rails-4842a8377644/actionpack/lib/action_dispatch/routing/route_set.rb, count: 7112
memsize: 1558248, /Users/kbrock/.gem/ruby/2.2.3/gems/journey-1.0.4/lib/journey/parser.rb, count: 38884
memsize: 1415448, /Users/kbrock/.gem/ruby/2.2.3/gems/mime-types-1.25.1/lib/mime/types.rb, count: 25410
memsize: 838943, /Users/kbrock/.gem/ruby/2.2.3/bundler/gems/rails-4842a8377644/activerecord/lib/active_record/base.rb, count: 1587
memsize: 837552, /Users/kbrock/.gem/ruby/2.2.3/bundler/gems/rails-4842a8377644/actionpack/lib/action_dispatch/routing/mapper.rb, count: 9775
memsize: 573170, poparser.ry, count: 9544
memsize: 568560, /opt/rubies/ruby-2.2.3/lib/ruby/2.2.0/rubygems/basic_specification.rb, count: 7060
master:
memsize: 15982987, /Users/kbrock/.gem/ruby/2.2.3/gems/activesupport-4.2.3/lib/active_support/dependencies.rb, count: 38136
memsize: 14126277, , count: 77574
memsize: 8778265, /opt/rubies/ruby-2.2.3/lib/ruby/2.2.0/rubygems/specification.rb, count: 142821
memsize: 3884879, /Users/kbrock/.gem/ruby/2.2.3/gems/mime-types-2.6.1/lib/mime/type.rb, count: 32304
memsize: 2734001, /Users/kbrock/.gem/ruby/2.2.3/gems/json-1.8.3/lib/json/common.rb, count: 24338
memsize: 2408812, /Users/kbrock/.gem/ruby/2.2.3/gems/actionpack-4.2.3/lib/action_dispatch/routing/route_set.rb, count: 12466
memsize: 1754440, /Users/kbrock/.gem/ruby/2.2.3/gems/actionpack-4.2.3/lib/action_dispatch/journey/parser.rb, count: 43797
memsize: 1346140, /Users/kbrock/.gem/ruby/2.2.3/gems/actionpack-4.2.3/lib/action_dispatch/journey/visitors.rb, count: 30805
memsize: 1045372, /Users/kbrock/.gem/ruby/2.2.3/gems/activerecord-4.2.3/lib/active_record/base.rb, count: 1934
memsize: 958991, /Users/kbrock/.gem/ruby/2.2.3/gems/actionpack-4.2.3/lib/action_dispatch/routing/mapper.rb, count: 11066
memsize: 898511, /Users/kbrock/.gem/ruby/2.2.3/gems/psych-2.0.13/lib/psych/visitors/to_ruby.rb, count: 6108
memsize: 860549, /Users/kbrock/.gem/ruby/2.2.3/gems/sass-3.4.16/lib/sass/engine.rb, count: 1362
memsize: 823502, /opt/rubies/ruby-2.2.3/lib/ruby/2.2.0/rubygems/basic_specification.rb, count: 11404
memsize: 711162, /Users/kbrock/.gem/ruby/2.2.3/gems/activesupport-4.2.3/lib/active_support/core_ext/module/delegation.rb, count: 1095
memsize: 667198, /Users/kbrock/.gem/ruby/2.2.3/gems/mime-types-2.6.1/lib/mime/types/cache.rb, count: 7524
memsize: 571332, poparser.ry, count: 9506
memsize: 548216, /Users/kbrock/.gem/ruby/2.2.3/gems/ruby_parser-3.7.0/lib/ruby21_parser.rb, count: 353
heh, curious if I bump the mime-types version in 5.4 how much it would change :)
hmm, action mailer pegs to 1.16 - thought a few others did as well
Keenan Brock
@kbrock
Oct 01 2015 19:57
so I was hoping to load our app using ruby 2.2
Alex Krzos
@akrzos
Oct 01 2015 19:58
So last appliance died after I tried to get allocation_tracer running
so new appliance now
5.5.0.3
Matthew Draper
@matthewd
Oct 01 2015 19:58
@kbrock in 5.4 / botvinnik? Good luck :)
Keenan Brock
@kbrock
Oct 01 2015 19:59
I just want it to boot
well, I got it to boot with new ruby/old rails, now I want to updgrade mime-types / rails and what ever minimal changes I need to make, to see where all that memory is used
Matthew Draper
@matthewd
Oct 01 2015 20:00
Isn't that what allocation_tracer is for? ¯\_(ツ)_/¯
Keenan Brock
@kbrock
Oct 01 2015 20:02
5.4.z with new rails
emsize: 11243881, , count: 56064
memsize: 8943578, /Users/kbrock/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb, count: 17803
memsize: 3885063, /Users/kbrock/.gem/ruby/2.2.3/gems/mime-types-2.6.1/lib/mime/type.rb, count: 32306
memsize: 3559423, /opt/rubies/ruby-2.2.3/lib/ruby/2.2.0/rubygems/specification.rb, count: 51678
memsize: 2188967, /Users/kbrock/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/routing/route_set.rb, count: 11189
memsize: 2039033, /Users/kbrock/.gem/ruby/2.2.3/gems/json-1.8.3/lib/json/common.rb, count: 20560
memsize: 1560680, /Users/kbrock/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/journey/parser.rb, count: 38953
memsize: 1198156, /Users/kbrock/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/journey/visitors.rb, count: 27392
memsize: 1046825, /Users/kbrock/.gem/ruby/2.2.3/gems/activerecord-4.2.4/lib/active_record/base.rb, count: 1959
memsize: 856194, /Users/kbrock/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/routing/mapper.rb, count: 9884
memsize: 667198, /Users/kbrock/.gem/ruby/2.2.3/gems/mime-types-2.6.1/lib/mime/types/cache.rb, count: 7524
memsize: 660379, /Users/kbrock/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/core_ext/module/delegation.rb, count: 1018
memsize: 573170, poparser.ry, count: 9544
I had to comment out a few things, but that is telling
oof, added ovirt back in...
memsize: 13222474, , count: 84231
memsize: 8841175, /Users/kbrock/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb, count: 17317
memsize: 7013116, /opt/rubies/ruby-2.2.3/lib/ruby/2.2.0/rubygems/specification.rb, count: 109404
memsize: 3885063, /Users/kbrock/.gem/ruby/2.2.3/gems/mime-types-2.6.1/lib/mime/type.rb, count: 32306
memsize: 2188855, /Users/kbrock/.gem/ruby/2.2.3/gems/actionpack-4.2.4/lib/action_dispatch/routing/route_set.rb, count: 11187
memsize: 2039033, /Users/kbrock/.gem/ruby/2.2.3/gems/json-1.8.3/lib/json/common.rb, count: 20560
Alex Krzos
@akrzos
Oct 01 2015 20:11
Alright looking better for allocation_tracer
Alex Krzos
@akrzos
Oct 01 2015 20:16
@jrafanie Still can't get allocation_tracer running even after adding it to the Gemfile.lock
/opt/rh/cfme-gemset/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274:in `require': cannot load such file -- allocation_tracer (LoadError)
    from /opt/rh/cfme-gemset/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274:in `block in require'
    from /opt/rh/cfme-gemset/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:240:in `load_dependency'
    from /opt/rh/cfme-gemset/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274:in `require'
    from akrzos.rb:2:in `trace'
    from akrzos.rb:18:in `<top (required)>'
    from /opt/rh/cfme-gemset/gems/railties-4.2.4/lib/rails/commands/runner.rb:60:in `load'
    from /opt/rh/cfme-gemset/gems/railties-4.2.4/lib/rails/commands/runner.rb:60:in `<top (required)>'
    from /opt/rh/cfme-gemset/gems/railties-4.2.4/lib/rails/commands/commands_tasks.rb:123:in `require'
    from /opt/rh/cfme-gemset/gems/railties-4.2.4/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
    from /opt/rh/cfme-gemset/gems/railties-4.2.4/lib/rails/commands/commands_tasks.rb:90:in `runner'
    from /opt/rh/cfme-gemset/gems/railties-4.2.4/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
    from /opt/rh/cfme-gemset/gems/railties-4.2.4/lib/rails/commands.rb:17:in `<top (required)>'
    from bin/rails:4:in `require'
    from bin/rails:4:in `<main>'
Joe Rafaniello
@jrafanie
Oct 01 2015 20:18
did you gem install allocation_tracer ?
Alex Krzos
@akrzos
Oct 01 2015 20:18
nope
sounds like I got some installing to do
Joe Rafaniello
@jrafanie
Oct 01 2015 20:18
yeah, try that, we have to hack bundler a bit to get it to work… manually install the gem, add it to the Gemfile, add it to the Gemfile.lock
Alex Krzos
@akrzos
Oct 01 2015 20:19
@jrafanie gotcha, just a pain on an appliance since it doesn't come with all the rpms to install gems
but I'm used to pain at this point
Matthew Draper
@matthewd
Oct 01 2015 20:20
Is it impractical to work with the upstream appliance instead?
Joe Rafaniello
@jrafanie
Oct 01 2015 20:20
yeah, we have to fix that at some point
Matthew Draper
@matthewd
Oct 01 2015 20:20
That seems far more likely to be pliable
Alex Krzos
@akrzos
Oct 01 2015 20:21
I can work with upstream as long as qe's automation works with it generally, takes care of all the setup pain
+1 to a cli for provider setup someday
Keenan Brock
@kbrock
Oct 01 2015 20:21
@jrafanie do you think we load all of the localization files into memory? is it possible we have more translations in 5.5?
Joe Rafaniello
@jrafanie
Oct 01 2015 20:24
yeah @akrzos, we can make changes to upstream if needed to get it to work “nicely” with qe auomation
or automation
@akrzos what exact ruby version is running on 5.5?
is it 2.2.0?
ruby -v
Alex Krzos
@akrzos
Oct 01 2015 20:25
ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux]
Joe Rafaniello
@jrafanie
Oct 01 2015 20:26
ok, thanks
Alex Krzos
@akrzos
Oct 01 2015 20:29
FYI ruby 2.2 devel rpm is rh-ruby22-ruby-devel for 5.5, maybe save someone else some future pain
@jrafanie So still need Gemfile to have gem 'allocation_tracer' in addition to the Gemfile.lock edits
Joe Rafaniello
@jrafanie
Oct 01 2015 20:30
ah, yes, you needed ruby.h?
Alex Krzos
@akrzos
Oct 01 2015 20:30
yes to build the gem
@jrafanie here's the allocation trace from 5.5.0.3 - https://gist.github.com/akrzos/81b50d444ecc84086bc3
Joe Rafaniello
@jrafanie
Oct 01 2015 20:41
the memoist lines look interesting
Keenan Brock
@kbrock
Oct 01 2015 20:46
memoist could cause a memory leak
Joe Rafaniello
@jrafanie
Oct 01 2015 20:46
@akrzos there’s a hacky way to disable the generational GC, effectively make it do full GC... if we think we’re creating so may objects quickly that ruby keeps allocating memory faster than GC can reclaim them
export RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9 before you run your console that does the refresh
time will go through the roof but memory usage should be closer to what you saw with older rubies
Keenan Brock
@kbrock
Oct 01 2015 20:47
man - so I've been running the object space info, on 3 consecutive boots
and it is all over the place
Matthew Draper
@matthewd
Oct 01 2015 20:48
@kbrock STI parser caching?
Keenan Brock
@kbrock
Oct 01 2015 20:48
@matthewd this is 5.4 / ruby: 2.2.3 latest rails
Matthew Draper
@matthewd
Oct 01 2015 20:49
Ah, okay… so I don't really care, then :grinning:
Keenan Brock
@kbrock
Oct 01 2015 20:49
lol
I do have my eye on that
since ruby parser shows in the top 30
Joe Rafaniello
@jrafanie
Oct 01 2015 20:49
Rails console on 5.4.z was ~100 MB res locally, and ~170 res for master… with RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9, it’s ~130 MB with amster
or master
Keenan Brock
@kbrock
Oct 01 2015 20:50
(on master)
Keenan Brock
@kbrock
Oct 01 2015 20:57
@jrafanie tweak to your preinitializer.rb https://gist.github.com/kbrock/0521d8cd2ae8b931cbd5
much prettier output / very scannable
Joe Rafaniello
@jrafanie
Oct 01 2015 20:58
ok, will try it out
Keenan Brock
@kbrock
Oct 01 2015 20:58
do you have a gist for allocation_tracer ?
Joe Rafaniello
@jrafanie
Oct 01 2015 21:04
def trace
  require 'allocation_tracer'

  ObjectSpace::AllocationTracer.setup(%i{path line type class})
  result = ObjectSpace::AllocationTracer.trace do
    yield
    nil
  end

  result.each {|k, v|
    puts "#{v[0]}\t#{k[0]}:#{k[1]}\t#{k[2]}\t#{k[3]}\t#{k[4]}"
    # puts ([v[0]]+k).join("\t")
  }
end
the latter puts can be used if you exclude the “class” name in the .setup call
then you can do trace { something }
Jason Frey
@Fryguy
Oct 01 2015 21:04
jrafanie, to test what removing the translations would look like, just rm the .po files ;)
I'm curious how that will change you allocations
Joe Rafaniello
@jrafanie
Oct 01 2015 21:08
@kbrock trying your pretty version
Keenan Brock
@kbrock
Oct 01 2015 21:08
lol - I'll have it for you in a bit
I'm taking a break - on tenancy right now
Joe Rafaniello
@jrafanie
Oct 01 2015 22:08
FYI, the memoist results that @akrzos had here near the top will be fixed by @matthewd and Dennis’ changes in this ManageIQ/manageiq@e74859c, found in this ManageIQ/manageiq#4073
i recreated those results by running spec/models/manageiq/providers/vmware/infra_manager/refresher_spec.rb locally, and after git pull those memoist entries dropped off the list
it’s probably not a major improvement but we now clear the relationship cache after the bulk is done, instead of each time we add thigns