These are chat archives for ManageIQ/manageiq/performance

15th
Jan 2018
Joe Rafaniello
@jrafanie
Jan 15 2018 15:30
@NickLaMuro for fun, I tried your steps above to gdb in and try to make it leak... I'm 2 for 2 with causing the MIQ Server process to SIGABRT after running the first GC.start
:laughing:
Keenan Brock
@kbrock
Jan 15 2018 15:30
yay
@jrafanie crash and burn
Nick LaMuro
@NickLaMuro
Jan 15 2018 15:36
heh
Nick LaMuro
@NickLaMuro
Jan 15 2018 15:51
@jrafanie to be fair, I was using the Vagrant VM that this was already set up to work with... so honestly not sure what steps might be necessary when compiling ruby to make it work
possible that gdb needs to be present on the system when doing a ./configure for the compilation to be done properly
Joe Rafaniello
@jrafanie
Jan 15 2018 15:54
@NickLaMuro it doesn't like when I GC.start. It seems to not SIGABRT when I do teh sync_workers so I'm doing that now
Nick LaMuro
@NickLaMuro
Jan 15 2018 15:56
oh, weird
Keenan Brock
@kbrock
Jan 15 2018 16:14
proctitle? - very nice
@NickLaMuro did you narrow it down to which script is the main contender?
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:17
going to be looking at the results shortly
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:35
mmk, looks like it is not sys-proctable
and not "really" sys-uname either
Keenan Brock
@kbrock
Jan 15 2018 16:37
I'm curious what monkey patching it could be
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:38
what leaked a surprising amount was MiqSystem.num_cpus
Keenan Brock
@kbrock
Jan 15 2018 16:39
wha? but that is cached - no?
the require is probably a no-op. right?
@kbrock yeah
my thought is it "might" be something we override with require
Jason Frey
@Fryguy
Jan 15 2018 16:39
"leaked" or "created a lot of objects yet to be GC'd"?
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:40
Jason Frey
@Fryguy
Jan 15 2018 16:41
every 2 minutes should be plenty...that is crazy surprising
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:41
well, as @kbrock, it is cached in a class variable
so that really rules out that bit
Jason Frey
@Fryguy
Jan 15 2018 16:41
where?
Jason Frey
@Fryguy
Jan 15 2018 16:42
oh duh... i totally didn't see the ||=
haha
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:42
but there is the require "linux_admin" every time it is called, so curious if that is somehow causing an issue
Keenan Brock
@kbrock
Jan 15 2018 16:42
ugh. hard to not now suggest a better way to write that
Jason Frey
@Fryguy
Jan 15 2018 16:43
yeah
Keenan Brock
@kbrock
Jan 15 2018 16:43
so you'll write that as 2 different ways?
1 require up top
the other require inside?
Jason Frey
@Fryguy
Jan 15 2018 16:43
throw require "linux_admin" in the loop?
curious if you just do nothing at all in the loop if it still leaks :)
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:44

@Fryguy All the scripts are effectively using the same template:

NickLaMuro/miq_tools@7297024

and some don't leak at all
Jason Frey
@Fryguy
Jan 15 2018 16:44
keep up the good work! hoping you are drilling into the issue!
yup
Keenan Brock
@kbrock
Jan 15 2018 16:45
@NickLaMuro I had written off that method because it was cached
good job being thorough
huh
we are doing the delayed require more and more in our code
active support is overriding require?
@Fryguy are we messing with require? when we do the descendant stuff - or something else?
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:47
I am looking into that, but unless we require "require_with_logging" (optional), doesn't seem like it
Keenan Brock
@kbrock
Jan 15 2018 16:48
bundler version?
or rails version?

bundler version?
or rails version?

@kbrock are you asking, or are you thinking those are the culprits possibly?

Keenan Brock
@kbrock
Jan 15 2018 16:49
possible culprits
sorry -thanks for question
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:49
I know for sure that bundler undoes some things that rubygems does with require, but those should be in vanilla ruby (so people would have undoubtably run into them by now), and would have leaked when we tested it without config/environment
Keenan Brock
@kbrock
Jan 15 2018 16:50
yea, miq-kernel makes me a little nervous - even though it looks good enough
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:50
I am going to run a test with just doing require 'linux_admin' in a loop and see if that causes an issue
Keenan Brock
@kbrock
Jan 15 2018 16:50
if bundler is undoing stuff. they are assuming they are undoing rubygems stuff. maybe we threw them for aloop when we did this?
no require relative in linux_admin.rb
but require 'more_core_extensions/all'
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:52
well, after the first invocation, it should just return a true, if I am not mistaken
Keenan Brock
@kbrock
Jan 15 2018 16:52
yea
Nick LaMuro
@NickLaMuro
Jan 15 2018 16:53
(and not go further)
Keenan Brock
@kbrock
Jan 15 2018 16:53
agreed
Sys::Platform::IMPL is a constant - right?
would accessing that posisbly load stuff?
comes from sys-uname if I am not mistaken
which I tested on it's own as well
didn't "really" leak that much
well, I will say, if it is require, then that would pretty much cause a leak in almost all of the workers
we do the deferred require technique in pretty much all of the code base
Jason Frey
@Fryguy
Jan 15 2018 16:58
I don't think we mess with it, but Rails does
activesupport-5.0.6/lib/active_support/dependencies.rb:291
although we may have messed with load...checking
nope, I can't see that we've messed with load
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:03
yeah, same
but it definitely looks like that is the cause
Jason Frey
@Fryguy
Jan 15 2018 17:04
require in a loop you mean?
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:04
yeah
Jason Frey
@Fryguy
Jan 15 2018 17:04
which version of Ruby?
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:04
2.3.1
Jason Frey
@Fryguy
Jan 15 2018 17:04
can you try 2.3.2 (or > 2.3.1)
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:04
well, not on an appliance, but I can check it locally
Jason Frey
@Fryguy
Jan 15 2018 17:05
curious if this is fixed or something in a newer Ruby
k
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:05
sure
Joe Rafaniello
@jrafanie
Jan 15 2018 17:10
For "fun", I'm running 500.times { MiqWorker.status_update_all; sleep 5} in rails console (with workers running) on both malloc and jemalloc ruby 2.3.1 appliances and will see if they leak.
It seems odd that doing these iterations a GC.start is expected but doesn't prevent the leak whereas an explicit GC.start does
Oh well, food
Keenan Brock
@kbrock
Jan 15 2018 17:13
sorry - what did you just say about GC.start?
Jason Frey
@Fryguy
Jan 15 2018 17:13
that GC.start is seemingly not being called
because calling it manually "kills" the leak
Keenan Brock
@kbrock
Jan 15 2018 17:14
on our worker processes, we call GC every 15 minutes
Jason Frey
@Fryguy
Jan 15 2018 17:14
ah interesting :+1:
Keenan Brock
@kbrock
Jan 15 2018 17:14
not sure about server process
sad. not sure if you guys saw this: https://bugzilla.redhat.com/attachment.cgi?id=1381554
hmm
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:15
well, I don't have a kbrock user on my machine ;)
Keenan Brock
@kbrock
Jan 15 2018 17:15
lol
turkey. that is what the "hmm" was for. took me a bit to get it
Jason Frey
@Fryguy
Jan 15 2018 17:16
ugh..it's worse
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:34

@kbrock @Fryguy so right now, I have this:

https://github.com/NickLaMuro/miq_tools/blob/efed9bf/miq_server_leak_discovery/17_test_require_linux_admin.rb

running with ruby-2.3.3 on my local machine, and it seems to be leaking a Mb every 5 min or so

Keenan Brock
@kbrock
Jan 15 2018 17:41
no way
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:44
and removing the require 'config/environment' makes the leak go away
Keenan Brock
@kbrock
Jan 15 2018 17:44
no!
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:45
@kbrock you are making this sound like a bad horror movie...
Keenan Brock
@kbrock
Jan 15 2018 17:45
opposite
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:45
good horror movie?
Keenan Brock
@kbrock
Jan 15 2018 17:45
it was the "I just found the meaning of life"
no way!
Jason Frey
@Fryguy
Jan 15 2018 17:45
wut
Keenan Brock
@kbrock
Jan 15 2018 17:45
(aka yay)
Jason Frey
@Fryguy
Jan 15 2018 17:48
dumb question, but config/environment is manageiq's config/environment?
(i.e. not a dummy app)
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:50
yes, correct
I am running this from the context of within a ManageIQ dir
Jason Frey
@Fryguy
Jan 15 2018 17:51
and with what command line are you running it? (trying to duplicate locally, so we have two points of reproduction)
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:51
to be fair, my current test is with manageq:master, and the one I was doing on an appliance is a CFME 5.9.something, but they both seem roughly equivalent
Jason Frey
@Fryguy
Jan 15 2018 17:52
yeah, I am on master
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:52

@Fryguy Just pushed this:

NickLaMuro/miq_tools@a5e3b76

Which has a flag toggle in it

and examples on how I am running it
Jason Frey
@Fryguy
Jan 15 2018 17:53
ah thanks!
Keenan Brock
@kbrock
Jan 15 2018 17:53
@NickLaMuro you reproducing on mac? or just appliance?
Jason Frey
@Fryguy
Jan 15 2018 17:53
mac
it's what I have
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:53
well, both for me
I first checked on an appliance, but then ran it locally to confirm
when testing it locally, I am just watching ActivityMonitor, since I don't want to work another script to monitor the mem usage on OSX...
Jason Frey
@Fryguy
Jan 15 2018 17:54
@NickLaMuro You have a Mac or Linux?
Nick LaMuro
@NickLaMuro
Jan 15 2018 17:54
OSX
Jason Frey
@Fryguy
Jan 15 2018 17:54
ah Mac :)
Jason Frey
@Fryguy
Jan 15 2018 18:05
I am not seeing the same results...how long did you wait?
I'm wondering if maybe the GC is just never kicking in your tests because of the multiple Time.now.min checks (i.e. it's possible that it never trigger both conditionals if there's a race)
Nick LaMuro
@NickLaMuro
Jan 15 2018 18:06
well, it is still running for me, but it started pretty quick for me
Jason Frey
@Fryguy
Jan 15 2018 18:06
hmmm
let me try again
does it creep up over time, or is it up and down?
Nick LaMuro
@NickLaMuro
Jan 15 2018 18:09
locally, I have them running side by side:
  • one was with "config/environment", and it started at 141.6MB when I started, and is now sitting at ~150MB (ticks up 0.1MB every 30 sec or so)
  • one without "config/environment" (WITHOUT_MIQ_ENV=1), and that has held at 30.5MB the entire time
creeps up over time
Jason Frey
@Fryguy
Jan 15 2018 18:09
ok, I'll just let this run then
I am also seeing a slow creep of 0.1 every 30 secs or so, but waiting for a GC (which is every 2 mins, I believe)
Nick LaMuro
@NickLaMuro
Jan 15 2018 18:10
right, mine has been running for 30ish minutes now, so plenty of GC's have happened since then
Jason Frey
@Fryguy
Jan 15 2018 18:10
I am going to tweak it to puts a "."...just so I can be sure it's GCing
Jason Frey
@Fryguy
Jan 15 2018 18:17
I am not seeing it GC ever
trying again with this instead:
(puts "."; GC.start) if Time.now.to_i % 120 == 0
Jason Frey
@Fryguy
Jan 15 2018 18:22
interesting...the process still creeps up verrrrrry slowly
Nick LaMuro
@NickLaMuro
Jan 15 2018 18:31
(puts "."; GC.start) if Time.now.to_i % 120 == 0
For the record, I think that is going to GC on a very irregular interval.
@Fryguy I think my issue might be that I didn't ever initialize the do_gc variable
and that is mucking with things
Jason Frey
@Fryguy
Jan 15 2018 18:32
really? should be exactly at the top of every 2 minutes
Nick LaMuro
@NickLaMuro
Jan 15 2018 18:32
that assumes you land on Time.now properly
Jason Frey
@Fryguy
Jan 15 2018 18:32
(and that's what I'm seeing with my puts ".")
oh i do...the loop runs so fast I always hit it
Nick LaMuro
@NickLaMuro
Jan 15 2018 18:33
yeah, the idea with mine is that you can basically put any sleep less than 30.sec, and it should GC regularly
but yes, I can confirm that it wasn't doing that with what I had (seems like adding a do_gc = nil prior to the loop starting seems to make it work)
Jason Frey
@Fryguy
Jan 15 2018 18:34
letting this run...it's really strange
Nick LaMuro
@NickLaMuro
Jan 15 2018 18:35

it's really strange

Agreed, I am probably going to do this in gdb later, and try and pin point what the new memory being allocated is, and from where

Jason Frey
@Fryguy
Jan 15 2018 19:35
@NickLaMuro Here are my results using my changes:
2018-01-15 13:24:02 -0500: 177496064
2018-01-15 13:25:02 -0500: 177594368
2018-01-15 13:26:02 -0500: 177717248
2018-01-15 13:27:02 -0500: 177717248
2018-01-15 13:28:02 -0500: 177717248
2018-01-15 13:29:03 -0500: 177717248
2018-01-15 13:30:03 -0500: 177737728
2018-01-15 13:31:03 -0500: 177737728
2018-01-15 13:32:03 -0500: 177737728
2018-01-15 13:33:03 -0500: 177737728
2018-01-15 13:34:03 -0500: 177745920
2018-01-15 13:35:03 -0500: 177745920
2018-01-15 13:36:03 -0500: 177815552
2018-01-15 13:37:03 -0500: 177815552
2018-01-15 13:38:03 -0500: 177827840
2018-01-15 13:39:04 -0500: 177827840
2018-01-15 13:40:04 -0500: 177831936
2018-01-15 13:41:04 -0500: 177836032
2018-01-15 13:42:04 -0500: 177836032
2018-01-15 13:43:04 -0500: 177840128
2018-01-15 13:44:04 -0500: 177856512
2018-01-15 13:45:04 -0500: 177860608
2018-01-15 13:46:04 -0500: 177868800
2018-01-15 13:47:04 -0500: 177868800
2018-01-15 13:48:05 -0500: 177881088
2018-01-15 13:49:05 -0500: 177897472
2018-01-15 13:50:05 -0500: 177901568
2018-01-15 13:51:05 -0500: 177901568
2018-01-15 13:52:05 -0500: 177905664
2018-01-15 13:53:05 -0500: 177909760
2018-01-15 13:54:05 -0500: 177909760
2018-01-15 13:55:05 -0500: 177913856
2018-01-15 13:56:05 -0500: 177913856
2018-01-15 14:01:02 -0500: 177934336
2018-01-15 14:02:03 -0500: 177934336
2018-01-15 14:03:03 -0500: 177938432
2018-01-15 14:04:03 -0500: 177938432
2018-01-15 14:05:03 -0500: 177942528
2018-01-15 14:06:03 -0500: 177963008
2018-01-15 14:07:03 -0500: 177967104
2018-01-15 14:08:03 -0500: 177967104
2018-01-15 14:09:04 -0500: 177971200
2018-01-15 14:10:04 -0500: 177975296
2018-01-15 14:11:04 -0500: 177975296
2018-01-15 14:12:04 -0500: 177983488
2018-01-15 14:13:04 -0500: 177987584
2018-01-15 14:14:04 -0500: 177991680
2018-01-15 14:15:04 -0500: 177995776
2018-01-15 14:16:04 -0500: 177995776
2018-01-15 14:17:05 -0500: 177999872
2018-01-15 14:18:05 -0500: 178008064
2018-01-15 14:19:05 -0500: 178012160
2018-01-15 14:20:05 -0500: 178016256
2018-01-15 14:21:05 -0500: 178020352
2018-01-15 14:22:05 -0500: 178024448
2018-01-15 14:23:05 -0500: 178024448
2018-01-15 14:24:05 -0500: 178032640
2018-01-15 14:25:05 -0500: 178044928
so yeah, ~1MB over the hour
GCs happen about 2 seconds before every one of those
Nick LaMuro
@NickLaMuro
Jan 15 2018 19:58
Something I am proposing as a quick fix, and testing on a few appliances, to see if it helps with the MiqServer leaks
ManageIQ/manageiq-gems-pending#327
Currently running on three appliances, so I will report back on the results
that said, doing this on a massive scale is not really maintainable, so finding the proper issue with require (or other issues) is probably preferred
Joe Rafaniello
@jrafanie
Jan 15 2018 19:59
WAT
Nick LaMuro
@NickLaMuro
Jan 15 2018 20:00

WAT

@jrafanie ? Care to explain your outburst?

Joe Rafaniello
@jrafanie
Jan 15 2018 20:00
I'm so confused. So, it has nothing to do with reading /proc or shelling out? It's require?
Jason Frey
@Fryguy
Jan 15 2018 20:00
seems like it
at least on Ruby 2.3.1 and 2.3.3
Keenan Brock
@kbrock
Jan 15 2018 20:00
@NickLaMuro lol - read your second require
Jason Frey
@Fryguy
Jan 15 2018 20:00
@jrafanie ANNDD you have to load config/environment
Keenan Brock
@kbrock
Jan 15 2018 20:00
hmm - I'll comment inline
Nick LaMuro
@NickLaMuro
Jan 15 2018 20:00
@kbrock heh, thanks, got it
thankfully I didn't do that when doing this on the appliances directly
Keenan Brock
@kbrock
Jan 15 2018 20:01
yay
Nick LaMuro
@NickLaMuro
Jan 15 2018 20:02
@kbrock after writing that exact code four times in a row, looks like I done goofed
fixed
Joe Rafaniello
@jrafanie
Jan 15 2018 20:04
Are we leaking Ruby instruction sequences for linux admin?
Keenan Brock
@kbrock
Jan 15 2018 20:04
ooooooh
Jason Frey
@Fryguy
Jan 15 2018 20:04
curious if we can throw any gem into that loop
like maybe more_core_extensions or something
Nick LaMuro
@NickLaMuro
Jan 15 2018 20:05
worth checking
Keenan Brock
@kbrock
Jan 15 2018 20:06
I added something that loops over LiveObjects to see if objects are leaked (already knew this would be no but...) and true enough, the leak is there but the ruby memory is constant
there are no new ruby objects added and stuff
Nick LaMuro
@NickLaMuro
Jan 15 2018 20:09
worth being doubly sure on though, thanks for confirming!
Nick LaMuro
@NickLaMuro
Jan 15 2018 20:15

Pushed this for anyone that wants to try with another gem:

NickLaMuro/miq_tools@0d90367

But I am currently running it using "more_core_extensions", and it seems to still leak (a little early to truly call it though leaking though)

hmm... may have spoke too soon
yeah, looks like it stopped... interesting... though, I did change it to using a constant string... so I wonder if that made a difference
anyway, I am going to grab some lunch and relocate, back in a bit
Nick LaMuro
@NickLaMuro
Jan 15 2018 22:55
So interesting thing about the above: turns out that change actually seemed to stop the leak. Seems like how it was done before, it was passing a newly allocated string to require each time that it is called, instead of what happens after 0d90367, which passes the same instance of a string every time.
Keenan Brock
@kbrock
Jan 15 2018 23:18
Hmm. looking for where EsxHost is defined
ls /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-vmware-3b785e25da13/lib/manageiq/providers/vmware/
engine.rb   version.rb
I expected to see more vmware entries, like infra_manager
must be multiple of those gems or something
AAAHHH - s/lib/app hmm
thnks for previous post nick
Keenan Brock
@kbrock
Jan 15 2018 23:30
@dmetzger57 the leak may have gotten worse for 5.8 vs 5.9 - but the base memory profile is looking similar. so if we solve this leak (aka world peace) then the app should be great
Nick LaMuro
@NickLaMuro
Jan 15 2018 23:31
#KeenanforMissUniverse2018
Keenan Brock
@kbrock
Jan 15 2018 23:31
evm_MetricsCollectorWorker_pid_28780.png
@NickLaMuro ^ that is with storage stubbed
now running with perf capture disabled for all 3 methods (hopefully)
since it is a mixin, a little tricky to skip it
we'll see if it doesn't do anything but still has a leak
curious why other classes worker are not showing the leak as well, but I'm sticking with my "if it has a require it has a leak" assessment
/Users/kbrock/src/manageiq
[log_memory] manageiq $ ag --ruby " require *['\"]" app/ lib | wc -l
     120
Keenan Brock
@kbrock
Jan 15 2018 23:36
note the leading space in the search
mag --ruby "^  *require *['\"]" | wc -l
     329
where mag is multiple/miq ag - searching all manageiq-* repos (at least checked out that have that phrase
Keenan Brock
@kbrock
Jan 15 2018 23:41
ooh - getting topic lines - only 166 hits
looks like this paints a much nicer picture:
ag --ruby "^  *require *['\"]" ~/src/manageiq-* |grep require | sed 's/.*: *require//' | sed "s/[\"']//g" | sort | uniq -c | sort -nr
Nick LaMuro
@NickLaMuro
Jan 15 2018 23:47
yeah, figured we would have a lot of hits where this was happening
Keenan Brock
@kbrock
Jan 15 2018 23:47
um
it is a lot and all - but it is a long tail
half of the top 20 look like they should be removed
  12  yaml
   9  trollop
   9  builder
   6  openstack/openstack_event_monitor
   6  hawkular/hawkular_client
   5  miq-ipmi
   5  drb
   4  util/win32/miq-wmi
   4  uri
   4  linux_admin
   4  ipaddr
   3  zip/filesystem
   3  tempfile
   3  socket
   3  ovirt_metrics
   3  ovirt
   3  openstack/openstack_handle
   3  miq-process
   3  fileutils
   3  csv
drb, uri, socket, tempfile, ipaddr, yaml, builder, trollop, csv, fileutils
they look like easy ones
Nick LaMuro
@NickLaMuro
Jan 15 2018 23:49
yeah, some of those could be called at the top just fine, you are right
Keenan Brock
@kbrock
Jan 15 2018 23:49
I did a blanket for search, and was totally rough
wasn't going for perfect, but wanted to know just how bad it was
I changed search to only hit lib and app dirs - 252 hits.
I mean - it is bad enough to be a problem. and we're seeing the trend to use that construct more, and we're seeing our memory bloating
just wonder if we can ease back from using that for stuff like yaml - which seems part of the standard library
Nick LaMuro
@NickLaMuro
Jan 15 2018 23:52
again, I think for stuff like 'yaml', at the top of the file would be fine
Keenan Brock
@kbrock
Jan 15 2018 23:52
having more fun
I did the same search but with ^require
the gems may be botching this up but...
the top 2 are yaml and linux_admin
# global ^requires
   9  yaml
   8  linux_admin
   7  thread
   7  awesome_spawn
   6  uri
   6  sys-uname
   6  manageiq/network_discovery/port_scanner
   6  fileutils
   5  ostruct
   5  openssl
   5  active_metrics/connection_adapters/abstract_adapter
   4  util/runcmd
   4  time
   4  sync
   4  rexml/document
   4  pathname
   4  mount/miq_generic_mount_session
   4  miq-system
   4  logger
   4  ancestry
so those embedded require constructs are actually doing nothing (assuming that the top level file is required)
hmm those counts don't look right. this is very rough
but it would be amusing if linux_admin were required already in global, and our embedded require was only leaking - not even requiring
um - global - meaning it is at the top of a file in the top name space / not in a method.
embedded meaning they are embedded in a def method
ok, I'm going to hit pause for a little as my vm tells me the memory profile of the NOP for the collector
Nick LaMuro
@NickLaMuro
Jan 15 2018 23:57
:wave: