These are chat archives for ManageIQ/manageiq/performance

27th
Oct 2015
Matthew Draper
@matthewd
Oct 27 2015 01:34
ManageIQ/manageiq#5135 :saxophone:
Looks much less impressive without numbers to back it up, but I'm only running on small, so I can only really claim "they seemed to get a bit smaller ¯\_(ツ)_/¯"
Oleg Barenboim
@chessbyte
Oct 27 2015 01:40
awesome @matthewd -- let's see if we can enlist the others ( @dmetzger57, @jrafanie ) to help with the measurement of speed and memory
Dennis Metzger
@dmetzger57
Oct 27 2015 01:50
@chessbyte I'll give it a spin, I'm running tonight's 8:50 build through tests right now
Alex Krzos
@akrzos
Oct 27 2015 14:32
FYI, latest results on 5.5.0.7 I have of my benchmarks are on google docs for reference - https://docs.google.com/a/redhat.com/spreadsheets/d/1kwdGs1c2oiQe0JCUZSS2PURWFnXmflLSQSGx3Lwt7Mw/edit?usp=sharing
Timings, RSS/Virt Start/Total/Change + GC.stat Before/After per benchmark
Refresh RHEVM/VMware host had an issue in my benchmark code, but fixing it now
Dennis Metzger
@dmetzger57
Oct 27 2015 16:17
Here's a couple views of the the data from my VMware Small environment run last night
Total RSS Size (Small Env).pdf
RSS Footprint (Small Env).pdf
RSS Footprint 2 (Small Env).pdf
Joe Rafaniello
@jrafanie
Oct 27 2015 16:18
@Fryguy @matthewd, I have a few things sucking up memory on the refresh worker and I'm not sure how to fix them
Jason Frey
@Fryguy
Oct 27 2015 16:18
@dmetzger57 Would it be possible to not make these pdfs...they are a little tough to quickly view
Jason Frey
@Fryguy
Oct 27 2015 16:18
particular if I'm on my phone
actually Excel is bad for phone too :)
Joe Rafaniello
@jrafanie
Oct 27 2015 16:19
1) I have a database with 3000 vms from @akrzos large env
Alex Krzos
@akrzos
Oct 27 2015 16:19
@dmetzger57 interesting, thats with master git pulled to latest?
Dennis Metzger
@dmetzger57
Oct 27 2015 16:19
@Fryguy sorry about, jpeg next time
Jason Frey
@Fryguy
Oct 27 2015 16:19
thanks :)
Joe Rafaniello
@jrafanie
Oct 27 2015 16:19
these vms appear to be duplicated 5 times in memory and all of them are select * from vms...
Dennis Metzger
@dmetzger57
Oct 27 2015 16:19
@akrzos that was from last nights build
Matthew Draper
@matthewd
Oct 27 2015 16:20
@dmetzger57 what's ruby that's not evm?
Joe Rafaniello
@jrafanie
Oct 27 2015 16:21
I believe the some(all) vms must be in multiple folders via folder <- folder <- vm
Dennis Metzger
@dmetzger57
Oct 27 2015 16:21
@matthewd i just pulled out the ruby processes to see if there was any difference, that number + evm gives the total size of the application
Jason Frey
@Fryguy
Oct 27 2015 16:21
wow nice @dmetzger57
the after memory is so much better
Dennis Metzger
@dmetzger57
Oct 27 2015 16:21
i've got to go through the large (then medium) data next
but first, food :smile:
Matthew Draper
@matthewd
Oct 27 2015 16:22
@dmetzger57 but all of evm is ruby… I'm trying to understand the distinction
Alex Krzos
@akrzos
Oct 27 2015 16:22
hmm nice, ruby perhaps evm_watchdog + web ui/web services worker?
due to how the processes show up in ps perhaps?
Dennis Metzger
@dmetzger57
Oct 27 2015 16:23
@matthewd @akrzos right, ps calls out a few processes as "ruby" where the workers show up as, well workers, actually not ps but /proc representation which is where ps comes from
Oleg Barenboim
@chessbyte
Oct 27 2015 16:26
@matthewd can you explain ManageIQ/handsoap#2 -- are we veering away from handsoap maintained in https://github.com/unwire/handsoap ?
Matthew Draper
@matthewd
Oct 27 2015 16:26
@chessbyte as I understand it, we already have
Alex Krzos
@akrzos
Oct 27 2015 16:26
so the latest P/Rs in should address all workers then if the sum of the workers is actually less than 5.4
Oleg Barenboim
@chessbyte
Oct 27 2015 16:26
@matthewd seems that the upstream has not been touched since Dec 2013
Matthew Draper
@matthewd
Oct 27 2015 16:27
Ultimately I'd obviously like for that to not be the case, but for now… ¯\_(ツ)_/¯
Oleg Barenboim
@chessbyte
Oct 27 2015 16:27
so, abandoned project?
Alex Krzos
@akrzos
Oct 27 2015 16:28
@dmetzger57 which of the october 26th upstream builds did you use? I see two from yesterday
Oleg Barenboim
@chessbyte
Oct 27 2015 16:28
@matthewd our branch is 29 commits behind unwire:master.
Matthew Draper
@matthewd
Oct 27 2015 16:28
:neutral_face:
Oleg Barenboim
@chessbyte
Oct 27 2015 16:29
13 commits ahead, 324 commits behind unwire:master.
Jason Frey
@Fryguy
Oct 27 2015 16:29
we are so far away from real handsoap
even the first iteration was so far away :(
Oleg Barenboim
@chessbyte
Oct 27 2015 16:29
makes me sad
Jason Frey
@Fryguy
Oct 27 2015 16:29
That's what I was working on locally...trying to upstream what I can, and abandon the rest
Oleg Barenboim
@chessbyte
Oct 27 2015 16:30
add this to Tech Debt list
Jason Frey
@Fryguy
Oct 27 2015 16:30
yup
I was hoping it might make performance better / easier to debug if we used upstream
things like the debug issue I found was completely our changes :(
Oleg Barenboim
@chessbyte
Oct 27 2015 16:31
but upstream has not been updated since Dec 2013
Jason Frey
@Fryguy
Oct 27 2015 16:31
that just means it's stable :)
Matthew Draper
@matthewd
Oct 27 2015 16:31
I think people just don't do soap anymore: https://www.ruby-toolbox.com/categories/soap — which is hardly news
Jason Frey
@Fryguy
Oct 27 2015 16:31
it's doesn't do very much, so no need for it to change much...plus who uses SOAP
Oleg Barenboim
@chessbyte
Oct 27 2015 16:32
VMware, ServiceNow
there have to be others
but I get the message
Matthew Draper
@matthewd
Oct 27 2015 16:32
As I understand it, current VMware has a REST API
Oleg Barenboim
@chessbyte
Oct 27 2015 16:32
really???
Jason Frey
@Fryguy
Oct 27 2015 16:33
EDIT: I think people (who write Rails apps, and thus are the majority of Ruby users) just don't do soap anymore
Oleg Barenboim
@chessbyte
Oct 27 2015 16:33
if so, I would not mind abandoning this costly SOAP implementation
Jason Frey
@Fryguy
Oct 27 2015 16:33
if VMware has REST, we should investigate, but we have to remember we still support old versions
Matthew Draper
@matthewd
Oct 27 2015 16:33
I don't know enough about the many acronyms to state that with any confidence, but it sorta sounded like that
(but yeah, I assumed we'd stick with soap for now because back-compat)
Jason Frey
@Fryguy
Oct 27 2015 16:35
I think vCloud is REST, but vSphere is still SOAP
@matthewd don't excite me like that ;-)
Jason Frey
@Fryguy
Oct 27 2015 16:36
vCenter Orchestrator has REST and SOAP
Oleg Barenboim
@chessbyte
Oct 27 2015 16:36
I cannot imagine VMware investing a LOT of $$ to redo their SOAP API as REST
not only is it a SOAP API - it is more of a data API than a functional API
Joe Rafaniello
@jrafanie
Oct 27 2015 16:40
this is the soap stuff that fog uses for vmware: https://github.com/rlane/rbvmomi
Matthew Draper
@matthewd
Oct 27 2015 16:40
@jrafanie I think your core problem is that the Relationship stuff kills relations
Jason Frey
@Fryguy
Oct 27 2015 16:41
I was talking with @jrafanie yesterday about that
I'm thinking of introducing a block-based method something like transaction method
Oleg Barenboim
@chessbyte
Oct 27 2015 16:41
@jrafanie Latest commit 5dc0ca3 on Mar 20, 2012
Jason Frey
@Fryguy
Oct 27 2015 16:41
so within that block the caches are not cleared...maybe
Matthew Draper
@matthewd
Oct 27 2015 16:41
@chessbyte looks like it now lives at https://github.com/vmware/rbvmomi
Jason Frey
@Fryguy
Oct 27 2015 16:41
another option is to just have the add_children code modify the caches instead of clearing them
with respect to the rbvmomi, I remember we investiagated that
Oleg Barenboim
@chessbyte
Oct 27 2015 16:42
@matthewd This is a side project of a VMware employee and is entirely unsupported by VMware.
good to see VMware put their weight behind it, NOT
Jason Frey
@Fryguy
Oct 27 2015 16:42
and it's good for doing things like operations
but you can't really do broker with it (didn't have proper waitForUpdates support, for example)
haha...that EDIT
Joe Rafaniello
@jrafanie
Oct 27 2015 16:43
this makes me :cry:
irb(main):002:0> VmOrTemplate.count
=> 3000
irb(main):003:0> all_vms = EmsFolder.all.collect(&:all_vms_and_templates).flatten.uniq; nil
=> nil
irb(main):004:0> all_vms.count
=> 3000
irb(main):005:0> ObjectSpace.each_object(Vm) {}
=> 12000
irb(main):006:0> ObjectSpace.each_object(ActiveRecord::Base) {}
=> 30150
irb(main):007:0> GC.start
=> nil
irb(main):008:0> ObjectSpace.each_object(Vm) {}
=> 3000
irb(main):009:0> all_vms.count
=> 3000
irb(main):010:0> ObjectSpace.each_object(ActiveRecord::Base) {}
=> 3000
Matthew Draper
@matthewd
Oct 27 2015 16:43
@jrafanie yeah, that's the thing I'm talking about
Joe Rafaniello
@jrafanie
Oct 27 2015 16:43
yes
Matthew Draper
@matthewd
Oct 27 2015 16:43
We should fix that
Jason Frey
@Fryguy
Oct 27 2015 16:43
why is it so many?
Joe Rafaniello
@jrafanie
Oct 27 2015 16:43
it kills relations and we have no ability to say .select(:id, :name)
Jason Frey
@Fryguy
Oct 27 2015 16:44
what do you mean by "kills relations"
Joe Rafaniello
@jrafanie
Oct 27 2015 16:44
I'm guessing because folders can be in folders and both can have the same vm
Jason Frey
@Fryguy
Oct 27 2015 16:44
you are just querying
oh yeah, you're "doing it wrong"
Matthew Draper
@matthewd
Oct 27 2015 16:44
@Fryguy it loads them, and turns them into arrays
Jason Frey
@Fryguy
Oct 27 2015 16:44
but is that real code, or just example
I would think in real code you wouldn't loop over folders like that
because they are nested
Joe Rafaniello
@jrafanie
Oct 27 2015 16:44
yes, it's real code
Jason Frey
@Fryguy
Oct 27 2015 16:45
the "right" way to do it is to preload the whole tree
Jason Frey
@Fryguy
Oct 27 2015 16:45
then walk it
EmsFolder.descendants_arranged
Joe Rafaniello
@jrafanie
Oct 27 2015 16:45
preload doesn't matter, you still get duplicates
Jason Frey
@Fryguy
Oct 27 2015 16:45
of course you do...because all will descend into the same objects
EmsFolder (1)
  EmsFolder (2)
    Vm (1)
Dennis Metzger
@dmetzger57
Oct 27 2015 16:46
@akrzos manageiq-vsphere-master-201510262000-ae49d71a2f.ova
Jason Frey
@Fryguy
Oct 27 2015 16:46
That Vm is under both folders
if you do EmsFolder.all, you get it twice...once for each folder
matthewd @matthewd wonders whether it'd be easier to relation-ize Relationship, or just convert to closure_tree
Jason Frey
@Fryguy
Oct 27 2015 16:47
cool...closure_tree is new to me
even so...Relationship has the ability to query it all in 1 shot
we're just not using it properly
@jrafanie, can you change your ObjectSpace counter to just call EmsFolder.descendants_arranged and see what the numbers are?
might be cleaner to just call that and prune the tree based on the criteria
then whatever's left in the tree are the vms we have to update
Matthew Draper
@matthewd
Oct 27 2015 16:52
Yeah… and I don't imagine teaching things like that to return a Relation instead would be that hard
Jason Frey
@Fryguy
Oct 27 2015 16:52
maybe
though the objects are from multiple tables, so it might be difficult without crazy joins
Joe Rafaniello
@jrafanie
Oct 27 2015 16:53

@jrafanie, can you change your ObjectSpace counter to just call EmsFolder.descendants_arranged and see what the numbers are?

before we preload and call all_vms_and_templates on each folder?

Matthew Draper
@matthewd
Oct 27 2015 16:53
Ah, then maybe a descendants_of_this_one_particular_class variant or something
Jason Frey
@Fryguy
Oct 27 2015 16:53
@matthewd We have that too
Matthew Draper
@matthewd
Oct 27 2015 16:53
(would be easier to relationize)
Jason Frey
@Fryguy
Oct 27 2015 16:54
EmsFolder.descendants_arranged(:of_type => VmOrTemplate)
(or something like that)
then yeah...that might be able to be relationized
@jrafanie No, I'm not talking with replacing the real code...just do it in console
I am just curious if that will keep the numbers down
I would expect only 3000 VMs, then 1 AR::Base for each Relationship/Folder/etc in the tree
Matthew Draper
@matthewd
Oct 27 2015 16:58
What's _arranged ?
Jason Frey
@Fryguy
Oct 27 2015 16:58
after querying the relavent nodes, which is an Array, it arranges them in a nested hash structure
useful for walking the tree
Matthew Draper
@matthewd
Oct 27 2015 16:59
I don't think we need that here, because we end up flattening?
Jason Frey
@Fryguy
Oct 27 2015 16:59
filtering is a lot simpler if it's arranged
because you can prune off entire branches
Matthew Draper
@matthewd
Oct 27 2015 17:01
Oh, right, because you're now walking the entire tree, not just the updated folders
Jason Frey
@Fryguy
Oct 27 2015 17:01
yes
Matthew Draper
@matthewd
Oct 27 2015 17:02
I guess we don't have a descendants_from_any_of_these_several_parents
Jason Frey
@Fryguy
Oct 27 2015 17:02
unfortunately no
but fortunately, there is only one parent root node
So, basically the code could be simplified with something like: extract_vms(prune_unchanged_folders(ems.ems_folder_root.descendants_arranged))
also, since all folders are clumped in the tree, prune_unchanged_folders can be optimized to stop walking when it sees something that's not an EmsFolder
so it wouldn't really walk all that much
As a total aside, this has me thinking about the Relationship tree desired state at the end of the refresh.
I think we can figure it all out in 1 query
update the tree by just modifying the relationships table directly
and then calling callbacks on all the objects where necessary
Alex Krzos
@akrzos
Oct 27 2015 17:07
@dmetzger57 I see, bummer no exact rhev build for that time frame but there is one a few hours older from it
Matthew Draper
@matthewd
Oct 27 2015 17:07
I think I need a deeper understanding of why the relationship tree is so generic
Alex Krzos
@akrzos
Oct 27 2015 17:07
I suppose I can do the git pull on both vmdb/appliance to get it to closest to upstream
Jason Frey
@Fryguy
Oct 27 2015 17:08
the reason we are calling add_vm, remove_vm etc is 1) for logging, which is a dumb reason, honestly and 2) some callbacks have minor side effects like setting flags (have to double check that all again)
Matthew Draper
@matthewd
Oct 27 2015 17:08
(providing for arbitrary non-self-referential structures, rather than just focusing on the recursive stuff, and leaving the rest to "normal" associations)
Jason Frey
@Fryguy
Oct 27 2015 17:09
@matthewd Like which method is confusing? (or needs clarification?)
Matthew Draper
@matthewd
Oct 27 2015 17:10
The concept. I see the advantage to using Relationship in the Folder-Folder link… but not in the Folder-Vm link.
Jason Frey
@Fryguy
Oct 27 2015 17:10
Relationship is really for creating a tree structure against different model types, and querying large parts of the tree structure in as little queries as possible
Matthew Draper
@matthewd
Oct 27 2015 17:10
Why not just a regular old Folder habtm Vm?
Jason Frey
@Fryguy
Oct 27 2015 17:11
because it's expensive query-wise to walk
if you have 100 folders, that's 100 queries
if you have a tree, it's 3 queries
Matthew Draper
@matthewd
Oct 27 2015 17:11
Right… so I think Relation fixes that
Jason Frey
@Fryguy
Oct 27 2015 17:12
normal ancestry does use relations, btw
but how does Relation solve that without joins
Matthew Draper
@matthewd
Oct 27 2015 17:12
If you have a Folder#descendant_folders method that returns a relation, then Folder has_many descendant_vms through descendant_folders source vms
my_folder.descendant_vms <— one query, all the things
Jason Frey
@Fryguy
Oct 27 2015 17:13
but that's the simplification
Joe Rafaniello
@jrafanie
Oct 27 2015 17:13
irb(main):002:0> tree = EmsFolder.first.descendants_arranged(:type => VmOrTemplate); nil
=> nil
irb(main):003:0> ObjectSpace.each_object(VmOrTemplate) {}
=> 3000
irb(main):004:0> ObjectSpace.each_object(ActiveRecord::Base) {}
=> 9286
Jason Frey
@Fryguy
Oct 27 2015 17:13
how do I do a tree with folders, hosts, clusters, vms, etc
resource_pools
@jrafanie yeah that seems about right to me
also if you checked the SQL it emits, it should only be like 3 queries or so
Joe Rafaniello
@jrafanie
Oct 27 2015 17:14
not sure how you'd walk all the updated folders though, you still end up with duplicates
Jason Frey
@Fryguy
Oct 27 2015 17:15
extract_vms(prune_unchanged_folders(ems.ems_folder_root.descendants_arranged))
Matthew Draper
@matthewd
Oct 27 2015 17:15
I don't see the advantage to allowing arbitrary nesting, as opposed to just the ones we know how to handle… this seems like the same discussion we had about storage, and the problems of building a data store atop a data store
Jason Frey
@Fryguy
Oct 27 2015 17:15
and extract_vms would pretty much be a copy/paste from that same file
Relationship handles more than just ems trees
unlike the storage case which right now just handles storage
(though admittedly, MiqSet can just be a normal has_many for nearly all of what it's used for)
Matthew Draper
@matthewd
Oct 27 2015 17:16
@jrafanie while copying extract_vms, you'll probably want flat_map
Dennis Metzger
@dmetzger57
Oct 27 2015 17:17
@akrzos i too did not find a RHEV build from last night either, so I'm only looking at VMware at the moment
Jason Frey
@Fryguy
Oct 27 2015 17:17
good eye @matthewd ...we should update that in the tree_builder code
@matthewd I'm all for changing the relationships to something better, but I really can't see how Relation will solve all the deep-tree problems
Matthew Draper
@matthewd
Oct 27 2015 17:18
Those all sound like they could benefit from a generic walk_tree helper, actually… seems like it could be less recursive
Jason Frey
@Fryguy
Oct 27 2015 17:18
I think I had that somewhere
or I had it and it turned out to be not much more than a .each call
Matthew Draper
@matthewd
Oct 27 2015 17:19
@Fryguy it definitely depends on what the model's trying to do — I think that's where I'm getting stuck, trying to separate how it works, from what its base objectives are
Jason Frey
@Fryguy
Oct 27 2015 17:19
yeah, I hear you
when you just want x has_many y relationships, I can sort of see how relationships seem to get in the way
when you want a "I need every x in this tree structure" method in as little queries as possible, then it really is worth it
Matthew Draper
@matthewd
Oct 27 2015 17:22
I guess my first question is: do we actually have trees where a given class will appear at multiple non-adjacent levels? (e.g., the equivalent of a folder inside a vm inside a folder?)
Jason Frey
@Fryguy
Oct 27 2015 17:23
I don't think so
but when I wrote it I didn't want to lock myself down in case we get a new provider with new rules
that's the other half of the problem...using relations locks you into a rigid structure and the providers are so different
but then again Host has_many Vms so, maybe we already broke that rule :)
Matthew Draper
@matthewd
Oct 27 2015 17:25
I think that returns to "there's no point being super-generic in how we store it, if the code still has to know how to do the thing with the thing"
But yeah, the general spectre of wanting to handle future-unknowns does complicate matters :confused:
Jason Frey
@Fryguy
Oct 27 2015 17:26
well, some code doesn't (like the tree builder)...it just builds what it sees more or less
or provisioning dialogs get the whole tree to filter UI elements...that's mostly generic
then provider specific code can walk the tree in specialized ways
some walk down folders, others walk down something else
Joe Rafaniello
@jrafanie
Oct 27 2015 17:32
note, this entire method is instantiating vm objects to call .id on them and compare [vm1, vm3] - [vm3]...
what do you think of putting this or something in EmsFolder so I can just look at ids?
  def vms_and_templates_ids
    Relationship.resource_ids(descendant_rels(:of_type => 'VmOrTemplate'))
  end
in other words, change instance methods to queue work for that self.id to instead be a class method that queues it for provided id parameter
Jason Frey
@Fryguy
Oct 27 2015 17:34
that doesn't instantiate VMs
or it shouldn't
Joe Rafaniello
@jrafanie
Oct 27 2015 17:35
yes, it doesn't
Jason Frey
@Fryguy
Oct 27 2015 17:35
but yeah, that could go in EmsFolder
Joe Rafaniello
@jrafanie
Oct 27 2015 17:35
that's what i'm suggesting instead of...
    added_vms = ems.vms_and_templates.where("created_on >= ?", update_start_time)

    # Create queue items to do additional process like apply tags and link events
    unless added_vms.empty?
      added_vm_ids = []
      added_vms.each do |v|
        v.post_create_actions_queue
        added_vm_ids << v.id
      end

      assign_ems_created_on_queue(added_vm_ids) if VMDB::Config.new("vmdb").config.fetch_path(:ems_refresh, :capture_vm_created_on_date)
    end
and...
    unless updated_folders.empty?
      descendant_ids_arranged
      updated_vms = updated_folders.collect(&:all_vms_and_templates).flatten.uniq - added_vms
      updated_vms.each(&:classify_with_parent_folder_path_queue)
    end
Matthew Draper
@matthewd
Oct 27 2015 17:36
@jrafanie but what about v.post_create_actions_queue?
Jason Frey
@Fryguy
Oct 27 2015 17:36
yeah, you could do that and call it only on the root folder
He can instatiate the VMs after he's filtered on them
Joe Rafaniello
@jrafanie
Oct 27 2015 17:36
  def post_create_actions_queue
    MiqQueue.put(
      :class_name  => self.class.name,
      :instance_id => id,
      :method_name => 'post_create_actions'
    )
  end
Jason Frey
@Fryguy
Oct 27 2015 17:37
or just make a clas method that avoids the instantiation
Joe Rafaniello
@jrafanie
Oct 27 2015 17:37
make a class method that does this
Jason Frey
@Fryguy
Oct 27 2015 17:37
You still need to filter on created_on
Joe Rafaniello
@jrafanie
Oct 27 2015 17:37
  def self.post_create_actions_queue(id)
    MiqQueue.put(
      :class_name  => self.name,
      :instance_id => id,
      :method_name => 'post_create_actions'
    )
  end
yes, @Fryguy.. all the filtering remains
Jason Frey
@Fryguy
Oct 27 2015 17:38
how about:
Joe Rafaniello
@jrafanie
Oct 27 2015 17:38
just select ids
Jason Frey
@Fryguy
Oct 27 2015 17:38
you can get the ids from the tree, then filter with a simple query
Oleg Barenboim
@chessbyte
Oct 27 2015 17:39
@matthewd @Fryguy - Alberto is asking for help in main manageiq room about class pre-loading
Jason Frey
@Fryguy
Oct 27 2015 17:39
Vm.where(:id => ems.ems_folder_root.vms_and_templates_ids).where("created_on >= ?", update_start_time).pluck(:id)
@chessbyte I think that's for @matthewd
I'm not sure on the descendant loader, but I thought it was fixed
Hoping this helps give more clarity on the relationships tree: https://gist.github.com/Fryguy/be705248227bcbf592b4
Jason Frey
@Fryguy
Oct 27 2015 17:44
Note that it is also possible to just call descendant_rels_arranged or, I think decendant_ids_arranged
to avoid hitting the other tables
Joe Rafaniello
@jrafanie
Oct 27 2015 17:48

Vm.where(:id => ems.ems_folder_root.vms_and_templates_ids).where("created_on >= ?", update_start_time).pluck(:id)

is that different than added_vms = ems.vms_and_templates.where("created_on >= ?", Time.at(0)).pluck(:id) ?

I don't see that instantiating Vm objects
Jason Frey
@Fryguy
Oct 27 2015 17:49
is vms_and_templates already cached?
Joe Rafaniello
@jrafanie
Oct 27 2015 17:50
i doubt it, it was a new console
I can leverage vms_and_template_ids method, just wasn't sure if there was a different reason you were thinking about
Jason Frey
@Fryguy
Oct 27 2015 17:57
so it's different on when you do the filtering
Matthew Draper
@matthewd
Oct 27 2015 18:08
@Fryguy are you okay with ManageIQ/handsoap#2 ?
Joe Rafaniello
@jrafanie
Oct 27 2015 18:18
@Fryguy @matthewd here's what I was thinking re: the instantiating of Vm objects just for their id: ManageIQ/manageiq@420df24
and here's the other one to limit the vm object to only the name and id columns for building the relationship tree: ManageIQ/manageiq@b971a73
testing both now
Jason Frey
@Fryguy
Oct 27 2015 19:19
@matthewd I'll review
Jason Frey
@Fryguy
Oct 27 2015 19:29
@jrafanie I still think it's too expensive query wise, but it's better than what's there now.
Because the problem is this: updated_folders.collect { query }which is one query per folder, which isn't necessary.
Joe Rafaniello
@jrafanie
Oct 27 2015 19:31
yeah
Dennis Metzger
@dmetzger57
Oct 27 2015 20:47
dang, forgot to post the vmware large environment charts from last night, its an age thing
Jason Frey
@Fryguy
Oct 27 2015 20:48
LOL
:older_man:
Oleg Barenboim
@chessbyte
Oct 27 2015 20:48
@dmetzger57 are we improving?
Dennis Metzger
@dmetzger57
Oct 27 2015 20:48
Large Env Sizes.png
Large Env Total RSS.png
Large Env RSS Breakdown.png
Large Env Process Breakout.png
Jason Frey
@Fryguy
Oct 27 2015 20:49
thank you for pngs ;)
Dennis Metzger
@dmetzger57
Oct 27 2015 20:49
anytime
Jason Frey
@Fryguy
Oct 27 2015 20:51
we got the broker lower!
Joe Rafaniello
@jrafanie
Oct 27 2015 20:51
thanks @dmetzger57
Jason Frey
@Fryguy
Oct 27 2015 20:51
the SAX parser should be even better!
Dennis Metzger
@dmetzger57
Oct 27 2015 20:51
so, in my large environment runs last night was showed a 10% growth in RSS memory used by the application, curious to see what @akrzos finds
Joe Rafaniello
@jrafanie
Oct 27 2015 20:51
Is the first 5 minutes of cap and u enough time to adequately measure those workers?
Jason Frey
@Fryguy
Oct 27 2015 20:51
I gotta get my damn proctitle change in...let me do that real quick
I wrote it and then never pushed it because I was trying to be fancy :/
Dennis Metzger
@dmetzger57
Oct 27 2015 20:52
no, we need a long term run for sure, this just gave a starting point though RSS was flat line during the entire 5 minutes (well essentially flat)
Joe Rafaniello
@jrafanie
Oct 27 2015 20:53
I don't remember when we do the bulk cap and u collection... I would think that's what we want to measure for generational GC "un-friendliness"
Dennis Metzger
@dmetzger57
Oct 27 2015 20:53
enough data to say we've gotten much better than we were a week ago
besides long term running, there's all the testing needed to see how all the features behave, this covers the first two that were shown to be rather bad in Master
Oleg Barenboim
@chessbyte
Oct 27 2015 20:54
that is fantastic
Jason Frey
@Fryguy
Oct 27 2015 20:54
yeah, performance of broker and refresh should be better as well
Oleg Barenboim
@chessbyte
Oct 27 2015 20:55
I am kind of afraid of introducing SAX parser now into the release
Jason Frey
@Fryguy
Oct 27 2015 20:55
like end to end time to refresh, broker priming time
Oleg Barenboim
@chessbyte
Oct 27 2015 20:55
better to get it burned in on master
Jason Frey
@Fryguy
Oct 27 2015 20:55
yeah, I want to wait on SAX parser
would feel more comfortable with specs behind it
Dennis Metzger
@dmetzger57
Oct 27 2015 20:55
@chessbyte fantastic as long as @akrzos find a similar improvement :smile: , I'm such a pessimist
Oleg Barenboim
@chessbyte
Oct 27 2015 20:55
@dmetzger57 good that you are a pessimist in your role
Matthew Draper
@matthewd
Oct 27 2015 20:56
I had assumed there was a VCR etc setup backing it… but yeah, if there are currently no specs on VimService, some caution is likely warranted
Jason Frey
@Fryguy
Oct 27 2015 20:56
what do you think of merging ManageIQ/manageiq#5094 first to get the short term benefits and then when we are comfortable with SAX parser we merge that
yeah, unfortunately there are 0 specs on that
there are things in the VMwareWebService/test directory, but they are not good for test
Matthew Draper
@matthewd
Oct 27 2015 20:57
(Though I do think it's fine — it went through the full small environment producing byte-for-byte identical output, and we could equally arrange such a comparison for large.)
Oleg Barenboim
@chessbyte
Oct 27 2015 20:57
VMware is still our bread/butter -- we need to be fantastic with that provider
Dennis Metzger
@dmetzger57
Oct 27 2015 20:57
@Fryguy i thought that was a good idea yesterday and still think thats a good (?safe?) change
Joe Rafaniello
@jrafanie
Oct 27 2015 20:58
i think 5094 is good to go, especially if we're not doing the SAX change yet... it makes no sense to keep around the xml string after it's parsed...
Jason Frey
@Fryguy
Oct 27 2015 20:59
let me un WIP
Oleg Barenboim
@chessbyte
Oct 27 2015 20:59
any concerns around merging it?
Jason Frey
@Fryguy
Oct 27 2015 20:59
@matthewd I trust that your code is good, but considering how vital it is to vmware working, I'd feel much more comfortable with specs
especially because I know of a few edge cases such as empty elements or Arrays that aren't Arrays that we might want to add specs for
^^ archaic knowledge of the VMware data being very inconsistent :(
Matthew Draper
@matthewd
Oct 27 2015 21:00
Yeah, I ran into some of those, I think
Jason Frey
@Fryguy
Oct 27 2015 21:00
it's one of the reasons we have that silly to_miq_a method
really upset I didn't push for automated tests there earlier :(
Matthew Draper
@matthewd
Oct 27 2015 21:01
I'm certainly not claiming we shouldn't write specs… and my interest in getting it into the release will depend entirely upon what the numbers look like
Jason Frey
@Fryguy
Oct 27 2015 21:02
I'm sure...you and I see eye to eye there :D
Matthew Draper
@matthewd
Oct 27 2015 21:04
@dmetzger57 you were going to do a large run of ManageIQ/manageiq#5135 with ManageIQ/handsoap#2 hacked in somehow, right?
Joe Rafaniello
@jrafanie
Oct 27 2015 21:04
Side note, I don't understand why we can't do a VCR setup for the vim stuff... we do it for the other providers going over http
Oleg Barenboim
@chessbyte
Oct 27 2015 21:05
I think all the other provider interactions populate our database without an intermediate process (VimBroker)
Jason Frey
@Fryguy
Oct 27 2015 21:05
unWIPed
Matthew Draper
@matthewd
Oct 27 2015 21:05
If the numbers are sufficiently shiny from that, I'll make sure we have enough spec coverage that we're confident to backport it for this release… if they're only mediocre, there's less hurry to polish it to mergeable
Oleg Barenboim
@chessbyte
Oct 27 2015 21:06
merged
Jason Frey
@Fryguy
Oct 27 2015 21:06
part of the problem is that only as of this past couple weeks did VCR go threadsafe
matthewd @matthewd needs a VimBroker 101 lesson at some stage, too, I think
Jason Frey
@Fryguy
Oct 27 2015 21:07
and the broker is threaded (though we can probably test parts of the broker without the thread stuff)
Joe Rafaniello
@jrafanie
Oct 27 2015 21:07
Right, I'm talking about the MiqVim layer, not broker
too much to deal with on the broker... I want to make sure the basic inventory stuff works reliably when we change things
Oleg Barenboim
@chessbyte
Oct 27 2015 21:07
I still think that MiqVim should be a thing
Jason Frey
@Fryguy
Oct 27 2015 21:07
yes, we can VCR it
Oleg Barenboim
@chessbyte
Oct 27 2015 21:07
and VimBroker should sit on top of it
Jason Frey
@Fryguy
Oct 27 2015 21:08
@chessbyte I think that's how it's actually built
I had a nice class diagram once
I guess I'll have to rebuild one :)
The confusing part is really the DMiqVim and the DRB stuff
Oleg Barenboim
@chessbyte
Oct 27 2015 21:09
but it would not break my heart if MiqVim were a separate repo and gem
and VimBroker would just require that
on ManageIQ/manageiq#5094, would setting response to nil achieve the same result?
Matthew Draper
@matthewd
Oct 27 2015 21:13
No; the caller still has a reference
Jason Frey
@Fryguy
Oct 27 2015 21:13
no, because the response object is inthe calling method
technically that line response.document is a noop oddly, and we could release it before that
buuuuuut...exactly 2 lines later we check if it's a fault response here: https://github.com/ManageIQ/handsoap/blob/handsoap_0_2_5/lib/handsoap/service.rb#L282
which loads the entire XML DOM
tl;dr...the patch code for making it lazy is completely useless
Matthew Draper
@matthewd
Oct 27 2015 21:17
Oh, that reminds me… @Fryguy I experimentally determined that SOAP Faults seemed to correspond with non-200 responses, so I just used that
Jason Frey
@Fryguy
Oct 27 2015 21:17
wow...that is cool
wonder if that's true all the time
(we can probably check the SDK documentation)