These are chat archives for ManageIQ/manageiq/performance

23rd
Oct 2015
Joe Rafaniello
@jrafanie
Oct 23 2015 14:25
So… @matthewd, while profiling a refresh, I found that with a 3000 vm vsphere, I had 6000 or so vm objects in memory… while @Fryguy thinks he knows why and this shoudl help decrease memory, it made me wonder if there’s a programmatic way to detect a missing “inverse_of” and find the originating association caller
I am running some refresh tests locally with some allocation tracing and indeed some examples do have duplicates of the model instance in ObjectSpace… but the sourcefile/line is pointing at AR where it allocates the object, not the association it’s coming from… Given an AR::Base object, can you tell what association it came from if it came from one?
diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb
index 3b63921..883089a 100644
--- a/spec/spec_helper.rb
+++ b/spec/spec_helper.rb
@@ -78,10 +78,22 @@ RSpec.configure do |config|

   config.before(:each) do
     EmsRefresh.debug_failures = true
+    if ENV["TRACE_ALLOCATIONS"]
+      GC.start
+      require 'objspace'
+      ObjectSpace.trace_object_allocations_start
+    end
   end
+
   config.after(:each) do
     EvmSpecHelper.clear_caches
+    if ENV["TRACE_ALLOCATIONS"]
+      ObjectSpace.trace_object_allocations_stop
+      EvmSpecHelper.duplicate_model_instance
+      ObjectSpace.trace_object_allocations_clear
+    end
   end
+
   if ENV["CI"] && ENV["TEST_SUITE"] == "vmdb"
     config.after(:suite) do
       require Rails.root.join("spec/coverage_helper.rb")
diff --git a/spec/support/evm_spec_helper.rb b/spec/support/evm_spec_helper.rb
index f1a5465..1eeb6a3 100644
--- a/spec/support/evm_spec_helper.rb
+++ b/spec/support/evm_spec_helper.rb
@@ -45,6 +45,25 @@ module EvmSpecHelper
     end
   end

+  def self.duplicate_model_instance
+    objects    = Hash.new {|h, k| h[k] = []}
+    duplicates = Set.new
+    ObjectSpace.each_object(ActiveRecord::Base) do |obj|
+      next unless obj.id
+
+      key = "#{obj.class.name.freeze}: #{obj.id}"
+      duplicates.add(key) if objects.key?(key)
+      objects[key] << "#{ObjectSpace.allocation_sourcefile(obj)}:#{ObjectSpace.allocation_sourceline(obj)}"
+    end
+
+    duplicates.each do |dupe_key|
+      $stderr.puts "Duplicate object found in memory: #{dupe_key}"
+      objects[dupe_key].each do |dupe|
+        $stderr.puts "  #{dupe}"
+      end
+    end
+  end
+
   def self.local_guid_miq_server_zone
     server = local_miq_server
     [server.guid, server, server.zone]
this is pretty useless:
Duplicate object found in memory: ManageIQ::Providers::Vmware::InfraManager::Vm: 745
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/inheritance.rb:61
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
Matthew Draper
@matthewd
Oct 23 2015 14:30
Maybe monkey-patch a thing into AR, to stash caller into an ivar on every record as it's instantiated? ¯\_(ツ)_/¯
Joe Rafaniello
@jrafanie
Oct 23 2015 14:32
i wonder if the activesupport notifications for sql can be used to find that information
Joe Rafaniello
@jrafanie
Oct 23 2015 14:45
oh man, I have to track this down…15+ instances of the same storage in ObjectSpace in the vmware refresh
Duplicate object found in memory: Storage: 176
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
  /Users/joerafaniello/.gem/ruby/2.2.3/bundler/gems/rails-532033180a9c/activerecord/lib/active_record/core.rb:114
Jason Frey
@Fryguy
Oct 23 2015 14:49
pair time?
Matthew Draper
@matthewd
Oct 23 2015 14:50
@jrafanie I don't remember what the auto-inverse_of stuff looks like, but maybe you can hack in some logging when it gives up?
Joe Rafaniello
@jrafanie
Oct 23 2015 14:51
ah, yeah, @matthewd, I can look into that
I was also going to see if I can use ObjectSpace.reachable_objects_from(obj) to see what assocations are reachable from each duplicate...
Jason Frey
@Fryguy
Oct 23 2015 14:52
@jrafanie See the duplicates method on more_core_extensions, if you want to save yourself a headache
Unless your code is different than that...hard to tell
Keenan Brock
@kbrock
Oct 23 2015 14:53
Do we think refresh would have less duplicate objects and less memory if we weren't invalidating the cache so often?
Jason Frey
@Fryguy
Oct 23 2015 14:54
that would just replace the objects, no?
Keenan Brock
@kbrock
Oct 23 2015 14:54
I've heard people mention concerns for not doing blowing away relations every time, but I don't see it
hmm
Madhu Kanoor
@mkanoor
Oct 23 2015 14:55
@dmetzger57 do you have some data on the Automate Worker. We were planning on reverting the commit that added the Automate Worker since we have not started sending work to it yet.
Matthew Draper
@matthewd
Oct 23 2015 14:57
@mkanoor what data do you need?
Oleg Barenboim
@chessbyte
Oct 23 2015 14:57
@mkanoor - we do need to revert that commit - simply set Automate Workers default count to 0
Jason Frey
@Fryguy
Oct 23 2015 14:58
I suggested the revert
so as not to carry all that unnecessary logic
because they would also have to hide the role in the UI amongst other things
Matthew Draper
@matthewd
Oct 23 2015 14:59
+100 not keeping code we don't [currently] use
Oleg Barenboim
@chessbyte
Oct 23 2015 14:59
maybe @Fryguy -- but I thought we really needed this worker
Jason Frey
@Fryguy
Oct 23 2015 14:59
apparently not?
Oleg Barenboim
@chessbyte
Oct 23 2015 14:59
confused as to why we created it and did not route any work to him
Jason Frey
@Fryguy
Oct 23 2015 14:59
It will have some benefits when we use it as it will leave generic workers open to do other work
also, it allows automate work to be routed to specific appliances
which is particularly important if there are locality issues with automate scripts (like only certain machines can hit the IPAM servers)
Dennis Metzger
@dmetzger57
Oct 23 2015 15:03
@mkanoor using about 250Mb of memory, I set the count to 0 for my last runs
Greg McCullough
@gmcculloug
Oct 23 2015 15:04
@chessbyte it was originally created as part of work to queue automate work which is currently synchronous, but that work has been deferred.
Oleg Barenboim
@chessbyte
Oct 23 2015 15:06
@gmcculloug @mkanoor so, is the idea to add it in the "D" release?
Madhu Kanoor
@mkanoor
Oct 23 2015 15:25
@chessbyte we had added and routed all the work to it but there were some hangs during Travis builds and we couldn't figure out why the tests were hanging, so that PR never got merged.
Oleg Barenboim
@chessbyte
Oct 23 2015 15:26
@mkanoor thanks for the clarification
Madhu Kanoor
@mkanoor
Oct 23 2015 15:26
@chessbyte ManageIQ/manageiq#3258
Oleg Barenboim
@chessbyte
Oct 23 2015 15:27
@mkanoor so why is this PR closed?
Madhu Kanoor
@mkanoor
Oct 23 2015 15:29
@chessbyte we could leave it as WIP
Madhu Kanoor
@mkanoor
Oct 23 2015 15:37
@chessbyte Instead of routing all work to Automate Worker we have a PR that routes the Provision related work which traditionally go thru the Queue to the Automate worker. ManageIQ/manageiq#4961
The problem started happening when we started routing all work to Automate UI, specifically the ones that come from the UI worker during dialog interactions.
Oleg Barenboim
@chessbyte
Oct 23 2015 15:39
yeah, unless you guys (@mkanoor and @gmcculloug) are against it, let's take our time and solve it cleanly in the "D" release. If you are in agreement, then disable/revert the current functionality
Greg McCullough
@gmcculloug
Oct 23 2015 15:52
@mkanoor I’m good with reverting it for now.
Madhu Kanoor
@mkanoor
Oct 23 2015 16:12
ok
Jason Frey
@Fryguy
Oct 23 2015 18:43
@matthewd What did you find with respect to the SAX parser?
Also, I'm surprised you got it working :)
Matthew Draper
@matthewd
Oct 23 2015 18:44
It actually slightly increased RSS :neutral_face:
Jason Frey
@Fryguy
Oct 23 2015 18:44
wow
that's not what I expected at all
Matthew Draper
@matthewd
Oct 23 2015 18:47
Keenan Brock
@kbrock
Oct 23 2015 18:49
@matthewd are there any pull parsers?
Jason Frey
@Fryguy
Oct 23 2015 18:50
what's a pull parser
Keenan Brock
@kbrock
Oct 23 2015 18:50
dom, they take the events and give you the document
Jason Frey
@Fryguy
Oct 23 2015 18:50
for XML?
Keenan Brock
@kbrock
Oct 23 2015 18:50
sax, you give them the controll, and they call you
pull parser, you retain control, and kinda generate the sax events yourself
aah. right. they call it StAX now
Jason Frey
@Fryguy
Oct 23 2015 18:51
interesting...never heard of it before...
<insert "the more you know" graphic here>
Keenan Brock
@kbrock
Oct 23 2015 18:52
MS made it popular ~2004 (?)
ends up being quicker than sax
and easier to use
sax is a single event loop
which can work, but is tricky, pull parsers allow you to have custom event loops - so you maintain less state
Jason Frey
@Fryguy
Oct 23 2015 18:53
yeah sax was always trickey to me because of the event stuff
felt weird to code against
Keenan Brock
@kbrock
Oct 23 2015 18:53
yes
also, sax always generates the events and objects
and always converts the attributes to objects
with a pull parser, you can skip over nodes, not converting to attribute hashes if you don't need them
so while metrics show it as similar to sax, in parcitce it is quicker
Matthew Draper
@matthewd
Oct 23 2015 18:55
Very-short-lived objects don't matter much to us ¯\_(ツ)_/¯
Keenan Brock
@kbrock
Oct 23 2015 18:55
aah. they call sax a "push streaming processor
aah - ok
aah. docs for java looks like it has been standardized for a while
Matthew Draper
@matthewd
Oct 23 2015 18:57
@Fryguy in my small/limited experiments, SAX was up to 10% faster, but left us with a 1-2% higher RSS… which, yeah, was obviously not what I expected either
Given how casually I'm dismissing the cost of building the attribute objects, I probably shouldn't be trying quite so hard to avoid allocations inside my state tracking… but even so, unless nokogiri/libxml is doing something avoidable & unpleasant, I'm not sure what else we could trim
Jason Frey
@Fryguy
Oct 23 2015 19:02
I was thikning that given the size of the XML we get, that objectifying the entire thing at once would create a much higher watermark than building objects as we went along
the latter being what I thought SAX would give us
Keenan Brock
@kbrock
Oct 23 2015 19:03
interesting, REXML DOM is 20% slower than libxml BUT REXML pull parser is only 2% slower
Matthew Draper
@matthewd
Oct 23 2015 19:03
Yeah… I think the hashes (and their contained super-heavy VimStrings) just far outweigh the actual parsed structure
Keenan Brock
@kbrock
Oct 23 2015 19:03
why do we have VimString again?
Jason Frey
@Fryguy
Oct 23 2015 19:03
VimStrings really should be all that big though...They are normal strings with 2 extra symbols
Keenan Brock
@kbrock
Oct 23 2015 19:04
is it possible to just not create those objects?
Jason Frey
@Fryguy
Oct 23 2015 19:04
That is, a VimString is really no bigger than it's associated regular String
need vim_type and xsi_type
Matthew Draper
@matthewd
Oct 23 2015 19:05
We seem to use those very rarely, compared to the amount of data we're retaining, though
Oleg Barenboim
@chessbyte
Oct 23 2015 19:07
yeah maybe a VimType and XsiType hash with a regular string as key?
or can the same string have different vim_type and xsi_type?
Matthew Draper
@matthewd
Oct 23 2015 19:08
From what I can see, we should be able to more selectively retain vim_type and xsi_type only on the ones we'll end up asking
Jason Frey
@Fryguy
Oct 23 2015 19:08
The same string can have different vim_type and xsi_type
@matthewd How would we tell the broker which ones we care about
maybe through the selector spec?
The broker needs it all to maintain inventory
Oleg Barenboim
@chessbyte
Oct 23 2015 19:09
The inventory cache should be on disk - Mongo, Redis, etc
at least as an option
Jason Frey
@Fryguy
Oct 23 2015 19:09
Those vim_type and xsi_type are needed when making requests back to vSphere
heck sqlite would even work :)
Keenan Brock
@kbrock
Oct 23 2015 19:10
I was thinking postgres (with memory only table) or sqlite
Matthew Draper
@matthewd
Oct 23 2015 19:10
Why do we fetch all this data from vsphere, then send it back?! :confused:
Keenan Brock
@kbrock
Oct 23 2015 19:10
there are some neat articles around sqlite and improving speed of it by 6,000x
Jason Frey
@Fryguy
Oct 23 2015 19:10
we don't send it back
Matthew Draper
@matthewd
Oct 23 2015 19:10
needed when making requests back
"heavy":
[2] pry(main)> ObjectSpace.memsize_of(VimString.new('xx', nil, nil))
=> 232
[3] pry(main)> ObjectSpace.memsize_of('xx')
=> 40
Jason Frey
@Fryguy
Oct 23 2015 19:11
That's for small strings, but that is interesting they are not embedded
try a big string (over 40 bytes)
When you request something else from vSphere, for example to stop a VM, you have to give it the VM MOR id with a vim_type and xsi_type
Oleg Barenboim
@chessbyte
Oct 23 2015 19:11
wow - 40 bytes just to hold a 2-character string??
Jason Frey
@Fryguy
Oct 23 2015 19:12
every Ruby object is 40 bytes
Oleg Barenboim
@chessbyte
Oct 23 2015 19:13
and almost 6x that for VimString
Jason Frey
@Fryguy
Oct 23 2015 19:13
yeah, it's not embedding strings, which is really interesting
Matthew Draper
@matthewd
Oct 23 2015 19:15
I think the ivars prevent embedding, maybe?
Jason Frey
@Fryguy
Oct 23 2015 19:15
oh maybe
if we can get the objects to be normal strings that would only really save memory on embeddable strings, right?
Matthew Draper
@matthewd
Oct 23 2015 19:16

you have to give it the VM MOR id with a vim_type and xsi_type

Sure… but do we ever need to know some_response['returnval']['locale'].xsiType == "SOAP::SOAPString"? :confused:

Jason Frey
@Fryguy
Oct 23 2015 19:16
curious what percentage of VimStrings are embeddable
yes
the parser makes that decision on certain values
Matthew Draper
@matthewd
Oct 23 2015 19:17
Is locale one of those values?
Jason Frey
@Fryguy
Oct 23 2015 19:17
example: app/models/manageiq/providers/vmware/infra_manager/refresh_parser.rb:863
more example: app/models/ems_folder.rb:139
Matthew Draper
@matthewd
Oct 23 2015 19:18
I'm not saying "we don't need any vimType data"… I'm saying "we don't need all the vimType data"
Jason Frey
@Fryguy
Oct 23 2015 19:18
oh I totally agree, but I'm questioning the savings
and also how would we inform the broker about when we do or don't need them
incidentally, we at least need it for every MOR (the id of every object from vim)
Oleg Barenboim
@chessbyte
Oct 23 2015 19:19
would converting the xsi_type and vim_type to a symbol for storage be more memory efficient?
Jason Frey
@Fryguy
Oct 23 2015 19:19
it's already a symbol
which is why I'm questioning the savings
Keenan Brock
@kbrock
Oct 23 2015 19:24
how many xsi_type, vim_type exist?
could you do @lookup = {id, [xsi_type, vmi_type]}
Are they like xml namespaces, or are they actually data values?
Jason Frey
@Fryguy
Oct 23 2015 19:25
There aren't that many...order of 100s if I recalls
Keenan Brock
@kbrock
Oct 23 2015 19:25
but 100s...
Jason Frey
@Fryguy
Oct 23 2015 19:25
100 symbols == nothing
Matthew Draper
@matthewd
Oct 23 2015 19:25
It's not about the values
Jason Frey
@Fryguy
Oct 23 2015 19:25
xsi_type is string like "SOAP::SOAPString"
There are finite set
vim_type is strings like "ManagedObjectReference"
Matthew Draper
@matthewd
Oct 23 2015 19:26
Err, it's not about the "type" values, it's about the fact that every individual value is dragging along knowledge of this type, when it doesn't care
Jason Frey
@Fryguy
Oct 23 2015 19:26
The parser doesn't care for the most part...other services do care
Matthew Draper
@matthewd
Oct 23 2015 19:26
And given that those individual values are mostly tiny strings, "just a couple of symbol references" is expensive
Jason Frey
@Fryguy
Oct 23 2015 19:27
is it?
100 symbols is roughly the same storage space as 100 embeddable and shared strings
Keenan Brock
@kbrock
Oct 23 2015 19:27
xml namespacing. that sure mucked up xml and basically killed it to the point that people use json/yaml.
Matthew Draper
@matthewd
Oct 23 2015 19:27
@Fryguy I'm not talking about that
Jason Frey
@Fryguy
Oct 23 2015 19:28
I don't disagree that I would prefer the parser only got normal strings/arrays/hashes when it asked for data
Keenan Brock
@kbrock
Oct 23 2015 19:28
all this extra qualification
Jason Frey
@Fryguy
Oct 23 2015 19:29
but I'm not sure how we can 1) ask the broker for what we need (maybe overload the selSpec) 2) prevent the broker from keeping it at all (maybe overload the PropMap)
Matthew Draper
@matthewd
Oct 23 2015 19:29
I feel like we have a much too generic API layer involved here, for the volume of data we're moving
Keenan Brock
@kbrock
Oct 23 2015 19:29
would it be cheaper to just string concat them with || ?
+1
Jason Frey
@Fryguy
Oct 23 2015 19:29
agreed...the broker is super-generic
probably doesn't need to be
Keenan Brock
@kbrock
Oct 23 2015 19:29
could the broker KNOW the vim_type when making the request?
not being passed in
Jason Frey
@Fryguy
Oct 23 2015 19:30
no
Keenan Brock
@kbrock
Oct 23 2015 19:30
:(
Jason Frey
@Fryguy
Oct 23 2015 19:30
that's part of the problem
so, say you want to add a Disk to a VM
you give it a VimString("vm-1234", :vimType => "ManagedObjectReference", :xsiType => "SOAP::SoapString")
Matthew Draper
@matthewd
Oct 23 2015 19:31
I'm presumably missing the forest for the trees, but when I grep for the xType attributes, I see ~12 lines in-app that use them in some way, and a bunch of stuff that seems to mostly be about maintaining them and/or setting them up for outgoing SOAP calls
Jason Frey
@Fryguy
Oct 23 2015 19:31
and also a VimString(:vimType => "ScsiSomethingOrOtherDisk", :xsiType => "SOAP::SoapString")
I am of course making that all up, but you get the idea :)
Keenan Brock
@kbrock
Oct 23 2015 19:32
y
Jason Frey
@Fryguy
Oct 23 2015 19:32
there is a column called ems_ref_obj that stores them
and they are passed along when making other calls, so you may have to grep for that
Matthew Draper
@matthewd
Oct 23 2015 19:32
Right.. so for each object, one particular type is a real attribute, as far as we're concerned
Jason Frey
@Fryguy
Oct 23 2015 19:32
might be in a generic path too
Matthew Draper
@matthewd
Oct 23 2015 19:33
We don't need the types of the rest of the attributes themselves
Jason Frey
@Fryguy
Oct 23 2015 19:33
from a future call perspective, yes
I'm not sure what the broker itself needs
to maintain it's own cache and for other call paths, like for operations or C&U
then there are the 12 or so cases that actually need to know the vimType
I'm not sure we ever need the xsiType though
or the other way around :)
Matthew Draper
@matthewd
Oct 23 2015 19:37
Assuming those aren't just usages of convenience (where there's another equally-valid thing we could inspect to make our decision), I'm totally fine with directly teaching the SAX/DOM parser to retain types for those particular keys
We should ultimately go to SAX, I think, because we should be able to stream right through, such that we never hold hashes for every known VM at once
Jason Frey
@Fryguy
Oct 23 2015 19:39
Do we have a broker heap dump available? I have an idea but want to verify something
yeah, that's what I was thinking about SAX
Joe Rafaniello
@jrafanie
Oct 23 2015 19:39
i can do it locally
Jason Frey
@Fryguy
Oct 23 2015 19:39
would be even better if handsoap could stream the XML string and we could subsequently stream that through the SAX parser
then we don't need the whole response document in memory at once
possibly
Matthew Draper
@matthewd
Oct 23 2015 19:40
Yeah.. I got most of the way there by hijacking the response as soon as it got stringified… but the old handsoap version we're on made it seem very difficult to go further
Jason Frey
@Fryguy
Oct 23 2015 19:40
:(
Matthew Draper
@matthewd
Oct 23 2015 19:40
I think I'd ditch handsoap entirely for these particular requests, and just do it manually
(maybe leaning on handsoap internals to construct the request, if that's worthwhile)
Jason Frey
@Fryguy
Oct 23 2015 19:41
I can't remember why @roliveri switched to handsoap
I think that's why he was using it
basically for the SOAP manipulation only
previously we were on soap4r
Greg McCullough
@gmcculloug
Oct 23 2015 19:41
@Fryguy handsoap was much faster/ligher weight than soap4r
Jason Frey
@Fryguy
Oct 23 2015 19:41
yeah
but it required @roliveri to roll his own a little bit
Matthew Draper
@matthewd
Oct 23 2015 19:42
Yeah… but it's still (obviously) a SOAPy API
Jason Frey
@Fryguy
Oct 23 2015 19:42
yeah
not much we can do there
Matthew Draper
@matthewd
Oct 23 2015 19:42
Well, we can e.g. stream the response
Jason Frey
@Fryguy
Oct 23 2015 19:42
yes
Matthew Draper
@matthewd
Oct 23 2015 19:43
.. which is definitely a departure from any strict reading of the SOAP spec
Jason Frey
@Fryguy
Oct 23 2015 19:43
I admit I don't know much about streaming from the HTTP perspective...does it require the serving side to be aware of it?
or can a client just say, give it to me in chunks and it Just Works :tm:
Matthew Draper
@matthewd
Oct 23 2015 19:44
I feel like we have several layers of formality built up around the SOAP service, and I instead want to pierce through them all, and just rip the bits we want out of some arbitrary XML we're receiving
The latter.
Jason Frey
@Fryguy
Oct 23 2015 19:44
cool
yeah, would love to avoid the SOAP formatlities
Matthew Draper
@matthewd
Oct 23 2015 19:45
I mean.. the server may send it in a whole go, in which case the client's going to end up buffering it all somewhere, but that part, you can't do much about
Jason Frey
@Fryguy
Oct 23 2015 19:45
yeah
Jason Frey
@Fryguy
Oct 23 2015 20:02
:scissors: on automate worker: ManageIQ/manageiq#5087
Matthew Draper
@matthewd
Oct 23 2015 20:03
So if we're doing that, does that mean we're now close enough on memory usage?
Jason Frey
@Fryguy
Oct 23 2015 21:30
@dmetzger57 ^^
I got the broker init_console working again
I'll get a PR in for that
oh wait maybe not...something is borked with the selector specs :(
@jrafanie you were seeing "getSelSpec: selector ems_refresh_storage not found" errors, right?
how did you bypass that?
Joe Rafaniello
@jrafanie
Oct 23 2015 21:34
hmm, let me look @Fryguy
ugh, put this in that script: ManageIQ::Providers::Vmware::InfraManager::Refresher.init_console(true)
sorry, ETOOMANYONEOFFSCRIPTS
Jason Frey
@Fryguy
Oct 23 2015 21:35
I'm not using your script
I'm just using rails console as the client
and I already did the init_console
and hacked the drb_port and all that other crap that for some reason is now required
Joe Rafaniello
@jrafanie
Oct 23 2015 21:36
so, you have the broker worker started and using console as the client?
Jason Frey
@Fryguy
Oct 23 2015 21:36
not a worker...I'm using the standalone broker
gems/pending/VMwareWebService/test/MiqVimBrokerServer.rb
Joe Rafaniello
@jrafanie
Oct 23 2015 21:37
oh, you’re on your own, it’s easier to find the broker with the worker
Jason Frey
@Fryguy
Oct 23 2015 21:37
that's the guts of the broker without all the worker overhead
Joe Rafaniello
@jrafanie
Oct 23 2015 21:37
;-)
Jason Frey
@Fryguy
Oct 23 2015 21:37
I can find the broker just fine
it's during priming I get that error...which I know you got around
Joe Rafaniello
@jrafanie
Oct 23 2015 21:39
yeah, I got around it by doing it through the worker and monkey patching the broker/runner
  def expected_broker_cache_scope
    :cache_scope_ems_refresh
  end
Jason Frey
@Fryguy
Oct 23 2015 21:53
thanks
hmmm...that doens't really help...there's some kind of bug with the broker
Oleg Barenboim
@chessbyte
Oct 23 2015 21:54
no more automate worker in Capablanca - merged ManageIQ/manageiq#5087
Jason Frey
@Fryguy
Oct 23 2015 22:00
on which enviroment were we seeing 1GB broker?
medium or large?
Joe Rafaniello
@jrafanie
Oct 23 2015 22:01
large
Jason Frey
@Fryguy
Oct 23 2015 22:01
ah...let me switch
Joe Rafaniello
@jrafanie
Oct 23 2015 22:01
locally it was 800 or so
Jason Frey
@Fryguy
Oct 23 2015 22:01
on Mac?
Joe Rafaniello
@jrafanie
Oct 23 2015 22:01
on an appliance with a full broker worker it was around 1 GB
Jason Frey
@Fryguy
Oct 23 2015 22:01
ok
Joe Rafaniello
@jrafanie
Oct 23 2015 22:01
yes, locally on mac, 800
Jason Frey
@Fryguy
Oct 23 2015 22:01
trying to get number for the pure broker server
without all the crap that we do in the worker
Joe Rafaniello
@jrafanie
Oct 23 2015 22:03
can we just merge this? I spent 15 minutes debugging why my ExtManagementSystem was still in ObjectSpace after GC
ManageIQ/manageiq#5025
realized we're still not running 4-2-stable
:cry:
Jason Frey
@Fryguy
Oct 23 2015 22:03
it's breaking Hakiri
or at least it appears that way...not sure what to do about it... Thoughts @chessbyte @matthewd ?
Matthew Draper
@matthewd
Oct 23 2015 22:04
:confused: the Hakiri page looks green
Jason Frey
@Fryguy
Oct 23 2015 22:04
does it?
Oh wtf
It was red last time I ran it
Joe Rafaniello
@jrafanie
Oct 23 2015 22:05
yeah, unfortunately, I have no idea, seems flaky
Jason Frey
@Fryguy
Oct 23 2015 22:05
merged
Joe Rafaniello
@jrafanie
Oct 23 2015 22:05
thank you
Matthew Draper
@matthewd
Oct 23 2015 22:05
Pull Request #5025 (commit #f8ef3f4) submitted 14 days ago
Jason Frey
@Fryguy
Oct 23 2015 22:06
14 days ago...weird
Matthew Draper
@matthewd
Oct 23 2015 22:06
Oh, I see… the commit was that old, even though the PR is younger
Jason Frey
@Fryguy
Oct 23 2015 22:06
ah...still, it reads funny
Joe Rafaniello
@jrafanie
Oct 23 2015 22:07
sounds about right, I have been running locally with 4-2-stable for testing for a while
keep cherry-picking the same commit to branches to do reliable testing
oh man
Jason Frey
@Fryguy
Oct 23 2015 22:07
?
Joe Rafaniello
@jrafanie
Oct 23 2015 22:07
@Fryguy you know the missing inverse_of I’m looking at...
Jason Frey
@Fryguy
Oct 23 2015 22:08
yeah?
Joe Rafaniello
@jrafanie
Oct 23 2015 22:08
ems_events on ems is one of them
can you imagine?
Jason Frey
@Fryguy
Oct 23 2015 22:08
only if it's called from both sides though, right?
Joe Rafaniello
@jrafanie
Oct 23 2015 22:08
yes
Jason Frey
@Fryguy
Oct 23 2015 22:08
Definitely we don't do that in refresh, but yeah, that could be an interesting one
Joe Rafaniello
@jrafanie
Oct 23 2015 22:09
if you called ems_events on both duped Ems, you could have millions of duplicated events
Jason Frey
@Fryguy
Oct 23 2015 22:14
you guys are going to love this....note: back of the envelope numbers VERY UNSCIENTIFIC using top
using a standalone broker I can watch the log to see what's going on...
preloading for large with the ems_refresh_cache_scope
Joe Rafaniello
@jrafanie
Oct 23 2015 22:15
yeah
Jason Frey
@Fryguy
Oct 23 2015 22:15
Before: ~32MB RSS
crap one sec
:P
After response XML received from handsoap: ~137MB
After parsing the document: ~970MB
After adding the objects to the inventory: ~1091MB
I will double check with inline code probes...like I said that's me watching the log and top at the same time
but as you can see it's ALL in parsing the document
the inventory memory is like 120MB extra
Matthew Draper
@matthewd
Oct 23 2015 22:19
NB, AFAIK, nokogiri doesn't create DOM nodes until they're needed
So, we are creating things other than the "result" hash during the parse
But yeah, that's why I want to :fire: all the VimStrings
Jason Frey
@Fryguy
Oct 23 2015 22:20
the XML size is 30_306_245 bytes
Matthew Draper
@matthewd
Oct 23 2015 22:21
(granted, I haven't confirmed it's the VimStrings in particular at fault… but within that block, there aren't many other things to blame..)
Jason Frey
@Fryguy
Oct 23 2015 22:21
but that's before we create all the VimStrings..I think
going to probe some more
I'll let you know what I find
Matthew Draper
@matthewd
Oct 23 2015 22:21
Hmm? parse is what makes the vimstrings
Jason Frey
@Fryguy
Oct 23 2015 22:22
sorry, by parse I meant the Handsoap parse
or whatever is being done initially after the handsoap response occurs
(like i said...need to probe exactly :) )
Keenan Brock
@kbrock
Oct 23 2015 22:28
before converting to sax, poke around to see if you can use an event pull rather than the sax event push.
yes, stax can use less memory, but the major win is ease of implementation. close to as easy as dom. you dont need the global state machine that sax requires. nor the string concatenation event gook
Jason Frey
@Fryguy
Oct 23 2015 22:29
does nokogiri have a pull parser (haven't checked)
Keenan Brock
@kbrock
Oct 23 2015 22:29
I think rexml and libxml do
Matthew Draper
@matthewd
Oct 23 2015 22:29
For our simple needs, I don't think the SAX parser ended up very complex ¯\_(ツ)_/¯
I've just had issues with sax in the past and seeing the articles tout why pull parsers were more convenient brought up previous nightmares.
but this is back in my java/xml days - probably > 10 years ago
Keenan Brock
@kbrock
Oct 23 2015 22:35
if we use nokogiri, can we also pass NOBLANKS, to cut down on the events?
oh well, happy weekend
Jason Frey
@Fryguy
Oct 23 2015 22:36
later @kbrock
Joe Rafaniello
@jrafanie
Oct 23 2015 22:48
have a good weekend all
Keenan Brock
@kbrock
Oct 23 2015 23:51
Matthew Draper
@matthewd
Oct 23 2015 23:52
Define leak?
It will grow, and not release the memory it's using
It's a cache
So if we were talking to a malicious server, it could cause a problem
But a rational server will give us a very limited set of type values over our/its lifetime