These are chat archives for ManageIQ/manageiq/performance

30th
Sep 2015
I have more benchmarks running throughout the day into the night I'll get more measurements on how much more memory usage is for vmware as well
I need to speed up the benchmarks or parallelize them
Dennis Metzger
@dmetzger57
Sep 30 2015 18:32
OK, @kbrock is taking a quick look at this
Alex Krzos
@akrzos
Sep 30 2015 18:33
@dmetzger57 @kbrock Thanks!
I also want to run the benchmarks with memory measurements against 5.3 so we can see Ruby 1.9.3 (5.3) vs Ruby 2.0.0 (5.4) vs Ruby 2.2 (5.5)
Alex Krzos
@akrzos
Sep 30 2015 18:39
rhevm-large is almost done and about 2x the size between 5.4 and 5.5
Dennis Metzger
@dmetzger57
Sep 30 2015 19:54

@Fryguy @matthewd could you take a look at the 1 line change I've made to finders.rb (in my fix_virtual_associations branch). The rollups were now blowing up because the resource (at least the MiqRegion case) is now an ActiveRecord::Relation. Is this change suffcient/acceptable? (I've run through a rollup without the ERROR with this change is in place)

Exception stack trace was:

[----] E, [2015-09-30T14:12:53.903281 #25001:af9998] ERROR -- : MIQ(MiqQueue#deliver) Message id: [2607], Error: [undefined method metric_rollups' for #<MiqRegion::ActiveRecord_Relation:0x0000000d669b28>] [----] E, [2015-09-30T14:12:53.903517 #25001:af9998] ERROR -- : [NoMethodError]: undefined methodmetric_rollups' for #<MiqRegion::ActiveRecord_Relation:0x0000000d669b28> Method:[rescue in deliver]
[----] E, [2015-09-30T14:12:53.903583 #25001:af9998] ERROR -- : /opt/rubies/ruby-2.2.3/lib/ruby/gems/2.2.0/gems/activerecord-4.2.4/lib/active_record/relation/delegation.rb:136:in method_missing' /opt/rubies/ruby-2.2.3/lib/ruby/gems/2.2.0/gems/activerecord-4.2.4/lib/active_record/relation/delegation.rb:99:inmethod_missing'
/var/www/miq/vmdb/app/models/metric/finders.rb:34:in find_all_by_range' /var/www/miq/vmdb/app/models/metric/finders.rb:14:inhash_by_capture_interval_name_and_timestamp'
/var/www/miq/vmdb/app/models/metric/rollup.rb:201:in rollup_child_metrics' /var/www/miq/vmdb/app/models/metric/rollup.rb:131:inblock in rollup_hourly'
/var/www/miq/vmdb/app/models/metric/rollup.rb:131:in each' /var/www/miq/vmdb/app/models/metric/rollup.rb:131:inrollup_hourly'
/var/www/miq/vmdb/app/models/metric/ci_mixin/rollup.rb:91:in `block (2 levels) in perf_rollup'

Matthew Draper
@matthewd
Sep 30 2015 19:55
:+1:
Alex Krzos
@akrzos
Sep 30 2015 20:14
@kbrock Added more details on VMware providers refresh also 2x memory usage in 5.5
Keenan Brock
@kbrock
Sep 30 2015 20:14
ugh
Joe Rafaniello
@jrafanie
Sep 30 2015 22:01
@akrzos have you been shown how to do heap dumps with allocation tracing on in ruby 2.2?
Joe Rafaniello
@jrafanie
Sep 30 2015 22:19
@akrzos maybe you can inject some logging on 5.5 to see some of the ObjectSpace information if you haven’t already
irb(main):001:0> require 'objspace'
=> true
irb(main):002:0> ObjectSpace.count_tdata_objects
=> {RubyVM::InstructionSequence=>32041, Proc=>3894, RubyVM::Env=>3772, :autoload_i=>516, :autoload=>74, Mutex=>83, Encoding=>100, ThreadGroup=>1, Binding=>3, Thread=>1, RubyVM=>1, :iseq=>1, Random=>1, ARGF.class=>1, Data=>1, Time=>1163, UnboundMethod=>4, :unknown=>5, OpenSSL::X509::Store=>1, FFI::StructLayout=>1, FFI::StructLayout::Array=>5, FFI::ArrayType=>5, FFI::FunctionType=>2, FFI::Function=>2, FFI::DynamicLibrary::Symbol=>2, FFI::DynamicLibrary=>1, Method=>10, :parser=>10, DateTime=>1, PG::BinaryDecoder::Float=>2, PG::BinaryDecoder::Boolean=>1, PG::BinaryEncoder::Boolean=>1, PG::BinaryDecoder::Bytea=>1, PG::BinaryEncoder::Bytea=>1, PG::BinaryDecoder::String=>1, PG::BinaryEncoder::String=>1, PG::BinaryDecoder::Integer=>3, PG::BinaryEncoder::Int8=>1, PG::BinaryEncoder::Int4=>1, PG::BinaryEncoder::Int2=>1, PG::TextDecoder::Date=>1, PG::TextEncoder::Date=>1, PG::TextDecoder::TimestampWithTimeZone=>1, PG::TextEncoder::TimestampWithTimeZone=>1, PG::TextDecoder::TimestampWithoutTimeZone=>1, PG::TextEncoder::TimestampWithoutTimeZone=>1, PG::TextDecoder::Float=>1, PG::TextEncoder::Float=>1, PG::TextDecoder::Boolean=>1, PG::TextEncoder::Boolean=>1, PG::TextDecoder::Bytea=>1, PG::TextDecoder::String=>1, PG::TextEncoder::String=>1, PG::TextDecoder::Integer=>1, PG::TextEncoder::Integer=>1, Object=>32, PG::TypeMapAllStrings=>1, :thread_shield=>3, :mutex=>3, JSON::Ext::Generator::State=>3, FFI::Type::Mapped=>1, FFI::Pointer=>1, FFI::Type::Builtin=>21, BigDecimal=>2}
irb(main):003:0> ObjectSpace.count_objects_size
=> {:T_OBJECT=>4997368, :T_CLASS=>6747520, :T_MODULE=>1454752, :T_FLOAT=>360, :T_STRING=>20733946, :T_REGEXP=>3029166, :T_ARRAY=>9196528, :T_HASH=>7106064, :T_STRUCT=>250480, :T_BIGNUM=>360, :T_FILE=>38048, :T_DATA=>39303517, :T_MATCH=>39200, :T_COMPLEX=>40, :T_RATIONAL=>35560, :T_SYMBOL=>44920, :T_NODE=>1162680, :T_ICLASS=>61560, :TOTAL=>94202069}
irb(main):004:0> ObjectSpace.memsize_of_all(String)
=> 20748214
irb(main):005:0> ObjectSpace.memsize_of_all
=> 89796947
Joe Rafaniello
@jrafanie
Sep 30 2015 22:54
the output from from this is really interesting, will need to look at this tomorrow
ruby -e 'require "objspace"; ObjectSpace.trace_object_allocations_start; require "./config/environment"; ObjectSpace.trace_object_allocations_stop; ObjectSpace.each_object { |o| puts(ObjectSpace.memsize_of(o).to_s + " bytes " + o.class.name + " " + ObjectSpace.allocation_sourcefile(o).to_s + ":" + ObjectSpace.allocation_sourceline(o).to_s) if ObjectSpace.memsize_of(o) > 10000  }' |sort -n
28664 bytes Hash :
28984 bytes RubyVM::InstructionSequence /Users/joerafaniello/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274
29624 bytes Hash /Users/joerafaniello/.gem/ruby/2.2.3/gems/mimemagic-0.3.0/lib/mimemagic/tables.rb:848
30248 bytes Hash /Users/joerafaniello/.gem/ruby/2.2.3/gems/rack-1.6.4/lib/rack/mime.rb:55
42312 bytes Hash /Users/joerafaniello/.gem/ruby/2.2.3/gems/mimemagic-0.3.0/lib/mimemagic/tables.rb:7
43840 bytes RubyVM::InstructionSequence /Users/joerafaniello/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274
45960 bytes Hash /Users/joerafaniello/.gem/ruby/2.2.3/gems/psych-2.0.15/lib/psych/visitors/to_ruby.rb:161
46313 bytes Regexp /Users/joerafaniello/.rubies/ruby-2.2.3/lib/ruby/2.2.0/rexml/parsers/baseparser.rb:58
57864 bytes MIME::Types::Container /Users/joerafaniello/.gem/ruby/2.2.3/gems/mime-types-2.6.1/lib/mime/types.rb:75
57920 bytes RubyVM::InstructionSequence /Users/joerafaniello/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274
70224 bytes RubyVM::InstructionSequence /Users/joerafaniello/.gem/ruby/2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:268
78825 bytes Regexp /Users/joerafaniello/.rubies/ruby-2.2.3/lib/ruby/2.2.0/rexml/parsers/baseparser.rb:114
93176 bytes MIME::Types::Container /Users/joerafaniello/.gem/ruby/2.2.3/gems/mime-types-2.6.1/lib/mime/types.rb:74
189432 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby19_parser.rb:1656
189432 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby19_parser.rb:801
200488 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby18_parser.rb:1685
200488 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby18_parser.rb:811
205072 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby20_parser.rb:1788
205072 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby20_parser.rb:869
210184 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby21_parser.rb:1830
210184 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby21_parser.rb:891
212000 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby22_parser.rb:1848
212000 bytes Array /Users/joerafaniello/.gem/ruby/2.2.3/gems/ruby_parser-3.7.1/lib/ruby22_parser.rb:900
234504 bytes FastGettext::GetText::MOFile /Users/joerafaniello/.gem/ruby/2.2.3/gems/fast_gettext-0.9.2/lib/fast_gettext/po_file.rb:14
366488 bytes Hash /Users/joerafaniello/.gem/ruby/2.2.3/gems/domain_name-0.5.24/lib/domain_name/etld_data.rb:5
1049920 bytes Thread :
Alex Krzos
@akrzos
Sep 30 2015 22:55
@jrafanie is objspace a gem already installed or will I have to add it?
@jrafanie I haven't done the heap dumps yet but Was thinking of that earlier this would be the perfect time to do so since the difference is on even the small providers
Joe Rafaniello
@jrafanie
Sep 30 2015 22:56
heading out, talk to you tomorrow
Matthew Draper
@matthewd
Sep 30 2015 23:05
@akrzos objspace is built in