These are chat archives for ManageIQ/manageiq/performance

8th
Mar 2017
Archit Sharma
@arcolife
Mar 08 2017 14:52
@dmetzger57 not used to talking on gitter everyday. I should enable notif switches for this.. btw, do you know or could you connect me to someone who would know a list of endpoints that CFME should be able to talk to, while adding an OSP overcloud provider?
example: Keystone Admin endpoint which is placed on the control plane network from OpenStack Director
Dennis Metzger
@dmetzger57
Mar 08 2017 14:54
Don’t have an anser for that :worried: let me ponder a bit and see if I can come up with someone to refer you to
Archit Sharma
@arcolife
Mar 08 2017 14:54
sure :)
Oleg Barenboim
@chessbyte
Mar 08 2017 14:55
@arcolife @dmetzger57 if you need OpenStack expertise on ManageIQ, @tzumainn is the lead on the parallel team that works on that integration
Dennis Metzger
@dmetzger57
Mar 08 2017 14:55
thanks @chessbyte
Archit Sharma
@arcolife
Mar 08 2017 14:56
thanks @chessbyte . Hello @tzumainn !
is he on this channel? couldn't autocomplete..
Tzu-Mainn Chen
@tzumainn
Mar 08 2017 14:58
@arcolife hi! yep, you need the keystone admin endpoint
ManageIQ will then look at the keystone catalog and use the endpoints it finds there to connect to various other services
so those would need to be exposed as well
Archit Sharma
@arcolife
Mar 08 2017 14:58
yeah.. @akrzos and I are running some CFME - OSP tests and are wondering which all networks we need to add to network-scripts
Tzu-Mainn Chen
@tzumainn
Mar 08 2017 15:01
@arcolife off the top of my head, you need public and admin keystone endpoints, and for the undercloud/infra provider you need mistral, glance, ceilometer, ironic, heat, ironic-inspector
Archit Sharma
@arcolife
Mar 08 2017 15:01
@tzumainn so we've added scripts for connecting to Gnocchi, keystone admin and I guess storage.
yes public ip
I see
Tzu-Mainn Chen
@tzumainn
Mar 08 2017 15:02
oh, and nova as well
Archit Sharma
@arcolife
Mar 08 2017 15:02
@tzumainn so all these aren't picked up from keystone catalog ?
Tzu-Mainn Chen
@tzumainn
Mar 08 2017 15:02
and for the overcloud, you'd want nova, neutron, cinder, gnocchi, ceilometer, heat, swift, glance
those are picked up from keystone catalog
Archit Sharma
@arcolife
Mar 08 2017 15:02
i see
Tzu-Mainn Chen
@tzumainn
Mar 08 2017 15:02
but ManageIQ still needs to be able to reach them
for the upcoming version of OSP you'd also want Panko
Archit Sharma
@arcolife
Mar 08 2017 15:03
for now we're onto OSP 10
Joe Rafaniello
@jrafanie
Mar 08 2017 15:05
am I the only one that gets hungry when someone mentions Gnocchi?
Archit Sharma
@arcolife
Mar 08 2017 15:05
let me fire up a new range of product named after vegan cuisines then.
s/vegan/whatever
Gregg Tanzillo
@gtanzillo
Mar 08 2017 15:06
I’m with you @jrafanie!
Oleg Barenboim
@chessbyte
Mar 08 2017 15:07
and Panko Breadcrumbs - mmm
Gregg Tanzillo
@gtanzillo
Mar 08 2017 15:07
Lunch can’t come soon enough!
Joe Rafaniello
@jrafanie
Mar 08 2017 15:07
nice @arcolife
Archit Sharma
@arcolife
Mar 08 2017 15:15
\m/
thanks @tzumainn
Joe Rafaniello
@jrafanie
Mar 08 2017 16:49
@dmetzger57 I wonder if this bz may explain the rise of memory usage of puma workers you saw :point_up: March 7, 2017 10:44 AM
Dennis Metzger
@dmetzger57
Mar 08 2017 16:53
@jrafanie probably not, before and during the test there is no UI activity against the appliance. Only thing that is done during the test is to add and refresh a provider using rails runner
Joe Rafaniello
@jrafanie
Mar 08 2017 16:53
do you have a breakdown of each type of puma worker?
Oleg Barenboim
@chessbyte
Mar 08 2017 16:54
I am still not convinced of the value we are getting by using puma vs. thin or other Rails-friendly app servers
Joe Rafaniello
@jrafanie
Mar 08 2017 16:56
well, I believe websocket/actioncable relies on puma... I don't know if you can run thin threaded for those two
I don't disagree but we've built features on top of puma and I don't know if those features are possible on thin
Dennis Metzger
@dmetzger57
Mar 08 2017 16:56
there’s 3 puma workers in the data set, PSS for each is ~190Mb
Joe Rafaniello
@jrafanie
Mar 08 2017 16:58
yeah, they seem to be about 30 MB PSS larger from 5.7 to 5.8
Oleg Barenboim
@chessbyte
Mar 08 2017 16:59
wow - 190mb?
Nick LaMuro
@NickLaMuro
Mar 08 2017 16:59
I made this claim yesterday (with no proof, to be fair), but I think it is worth repeating: I highly doubt that puma v.s. thin is the reason for change in PSS between versions. The cause for some threading issues with automate and such, sure, but not the cause of our memory woes
Oleg Barenboim
@chessbyte
Mar 08 2017 16:59
one day - we will all have 64gb or 128gb on our laptops
Joe Rafaniello
@jrafanie
Mar 08 2017 16:59
since they don't actually grow, I'd guess much of that 30 MB growth is new/updated gems/routes/etc.
Nick LaMuro
@NickLaMuro
Mar 08 2017 17:00
or the UI split possibly
(not sure how much overhead a Rails engine adds... for all I know, it might be nothing)
Oleg Barenboim
@chessbyte
Mar 08 2017 17:00
sigh
Keenan Brock
@kbrock
Mar 08 2017 17:01
question - I thought we changed our mime types gem to the mini one
Nick LaMuro
@NickLaMuro
Mar 08 2017 17:01
I believe with @dmetzger57 's tests, we can swap out thin for puma by changing some configs to confirm this theory
@kbrock I think that was just something that you wanted to do, but never followed through with
Keenan Brock
@kbrock
Mar 08 2017 17:02
lol
passing potato to fryguy
he is the one who mentioned it :)
Nick LaMuro
@NickLaMuro
Mar 08 2017 17:02
but my memory could be serving me incorrectly as well
Keenan Brock
@kbrock
Mar 08 2017 17:02
lol
I said +1 for someone ELSE to do ;) (which meant you? did I sign you up for it?) lol
ooh
Nick LaMuro
@NickLaMuro
Mar 08 2017 17:03
"voluntold"
Keenan Brock
@kbrock
Mar 08 2017 17:03
lol
the numbers I reported in standup were WRONG
I ran the numbers over 45 times - all agreed. but today it is showing that I sped up the widgets by 0%
@jrafanie were you the one who had a script to say how much memory each gem took up? maybe from scheems?
or maybe it was you nick
Nick LaMuro
@NickLaMuro
Mar 08 2017 17:05
definitely not me
but that sounds cool
Joe Rafaniello
@jrafanie
Mar 08 2017 17:05
yeah, there's something like that in derailed_benchmarks from scheems
Joe Rafaniello
@jrafanie
Mar 08 2017 17:05
we had something in the require with logging to do similar things on a require by require basis
Keenan Brock
@kbrock
Mar 08 2017 17:06
well running that on different versions may be useful. (again a :+1: for someone ELSE to try this)
that way we can narrow down memory usage from gems out of our control - at least at require time
Keenan Brock
@kbrock
Mar 08 2017 17:14
I'll give a case of beer (or a weeks worth of appetizers) to anyone who can turn assignee into a relation.
2 cases if you can generalize. we have dozens of these
class MiqAlertStatus < ApplicationRecord
  has_many :miq_alert_status_actions, -> { order "created_at" }, :dependent => :destroy
  virtual_column :assignee, :type => :string, :uses => :miq_alert_status_actions

  def assignee
    #miq_alert_status_actions.where(:action_type => %w(assign unassign)).last.try(:assignee)
    # simplified case that makes problem easier:
    miq_alert_status_actions.last.try(:assignee)
  end
end

# current fix
# MiqPreloader.preload(alert_statuses, :miq_alert_status_actions)
Oleg Barenboim
@chessbyte
Mar 08 2017 17:15
so, MiqAlertStatus has_one :assignee is what you want?
Keenan Brock
@kbrock
Mar 08 2017 17:15
yes
there are at least 5 stack overflows over the past years
very hard to define a has_one that is basically a has_many with a last / order by :created_at
Oleg Barenboim
@chessbyte
Mar 08 2017 17:17
why is it only the last status action?
maybe it assumes that the assignees of all the actions are the same and pulls the last one?
and why is an assignee on a status_action and not status?
Keenan Brock
@kbrock
Mar 08 2017 17:19
um
class MiqPolicy
  has_many                :policy_events

  def last_event
    policy_events.last.try(:created_on)
  end
end
heh, that one is probably a bug - not sure about order
Oleg Barenboim
@chessbyte
Mar 08 2017 17:19
yep - but that method is last_event
the other is not assignee_of_last_action, but simply assignee
Keenan Brock
@kbrock
Mar 08 2017 17:20
ok
module ComplianceMixin
  has_many :compliances, :as => :resource, :dependent => :destroy
  virtual_has_one :last_compliance, :class_name => "Compliance"
  def last_compliance
    return @last_compliance unless @last_compliance.nil?
    @last_compliance = if @association_cache.include?(:compliances)
                         compliances.max_by(&:timestamp)
                       else
                         compliances.order("timestamp DESC").first
                       end
  end
end
it is the pattern
our current solution is to download every compliances record
have a pr right now to download every miq_alert_status_actions ( ManageIQ/manageiq#13675 )
Joe Rafaniello
@jrafanie
Mar 08 2017 17:22
@dmetzger57 I just ran this locally with 5.7.0.17 vs. 5.8.0.2, the versions in your spreadsheet
12:17:16 ~/Code/manageiq (master) (2.3.3) + for x in 1 2 3; do; bin/rails r 'top = `/usr/bin/top -l 1 -pid #{Process.pid} | grep -E "ruby.+M"`; memory = top.split[7].to_i; puts memory'; done
** Using session_store: ActionDispatch::Session::MemCacheStore
144
** Using session_store: ActionDispatch::Session::MemCacheStore
146
** Using session_store: ActionDispatch::Session::MemCacheStore
149
That's RSS on osx, but these numbers are identical(~1-2MB) with master, 5.7.0.17, and 5.8.0.2 and ruby 2.3.3 so I'd imagine it's a ruby version / gem version thing. I did bin/update in all 3 scenarios so I got the latest of each gem according to our version constraints.
Keenan Brock
@kbrock
Mar 08 2017 17:23
@jrafanie units == MB?
Joe Rafaniello
@jrafanie
Mar 08 2017 17:23
yes, whatever osx top prints
If you compare the Gemfile.lock from those versions and ruby versions, I'd imagine we'd know the diff to concentrate on
In other words, I tested MIQ side and it appears fairly similar
Nick LaMuro
@NickLaMuro
Mar 08 2017 17:24
@jrafanie Should you have done a RAILS_ENV=production in there?
Joe Rafaniello
@jrafanie
Mar 08 2017 17:26
@NickLaMuro I don't think it matters in terms of requiring gems and our code. If master added a new/updated gem that bloats the process, you'd see it in dev mode too
Note, the PSS delta tab here is a delta sum across all processes so ones with more worker counts show up bigger. puma has 3 workers, reporting has 2, etc. It seems like each processes are around 25-50 MB PSS bigger.