These are chat archives for ManageIQ/manageiq/performance

14th
Jun 2016
Jason Frey
@Fryguy
Jun 14 2016 05:50
I just watched my PR and the "Fetching from git" piece was maybe 3 seconds...it flew by

for bundler cache total:

  • 5.67s attempting to download cache archive
  • 9.08s adding /home/travis/build/ManageIQ/manageiq/vendor/bundle to cache

= ~14.75s

Entire bundle install section 13.46s
Keenan Brock
@kbrock
Jun 14 2016 13:59
@Fryguy yay!
@Fryguy that was gems only - right? (not git based gems)
@akrzos is perf_capture_timer still an issue, looking at a number of BZs and unsure if we've solved this yet
Jason Frey
@Fryguy
Jun 14 2016 14:00
3 seconds was git based gems...entire bundler == all gems
3 was my estimate looking at the clock in the upper right :)
Keenan Brock
@kbrock
Jun 14 2016 14:01
what about the time to push the changes back to s3?
at the bottom
every time there is a git change, they have to rebuild the whole cache
Jason Frey
@Fryguy
Jun 14 2016 14:01
I didn't have a change in my PR, so it was nothing...I saw it in another PR but forgot what it was
Keenan Brock
@kbrock
Jun 14 2016 14:02
if there is a change in the git repo for any of the gems
Jason Frey
@Fryguy
Jun 14 2016 14:02
but if I recall it was something like 11 seconds
oh hmmm....I don't remember seeing anything in my PR
Keenan Brock
@kbrock
Jun 14 2016 14:02
if you keep running it, you won't see anything
Alex Krzos
@akrzos
Jun 14 2016 14:03
@kbrock I believe it is still an issue in 5.6
Keenan Brock
@kbrock
Jun 14 2016 14:03
but if anyone touches a gem's master, it changes and has to rebuild the cache again / push to s3
@akrzos thanks :(
Jason Frey
@Fryguy
Jun 14 2016 14:03
Looking at another PR it's 41.81s
Alex Krzos
@akrzos
Jun 14 2016 14:04
@kbrock I beleive the issue is really the growth in timing value in perf_capture_timer after a few captures
Keenan Brock
@kbrock
Jun 14 2016 14:04
@akrzos but the timing is now double - but will never grow above that
before, it used to grow forever
ok - thanks
Alex Krzos
@akrzos
Jun 14 2016 14:05
I believe it is reproducible with a large (3000vm, 1500 online (targets)) environment) now
Jason Frey
@Fryguy
Jun 14 2016 14:05
Then there is also 5.72s to execute bundle exec rake test:$TEST_SUITE:teardown
but I don't think we need that on CI, because who cares
it's really only for local test runs that might wreck your db
@jrafanie Thoughts ^?
Joe Rafaniello
@jrafanie
Jun 14 2016 14:14
Is that 41 seconds for each suite?
Jason Frey
@Fryguy
Jun 14 2016 14:15
yes
Joe Rafaniello
@jrafanie
Jun 14 2016 14:16
weird, maybe I don't understand the caching of gems but I thought it's done once for a full test run
at least for the rails based suites
Keenan Brock
@kbrock
Jun 14 2016 14:16
each build is independent
Joe Rafaniello
@jrafanie
Jun 14 2016 14:16
what's the point of caching then?
Keenan Brock
@kbrock
Jun 14 2016 14:16
they download the bundled gems
they run bundle
or bundle update (what ever)
if it changes they upload
if nothing changed, then nothing gets updated
if they are in parallel and there is a little duplicate work, you'll see multiple uploads of basically the same stuff
the next run, the gems should be up to date
but if we bundle update or we point to a git repo - that is changing (even if we are pointing to a specific branch) then the bundle changes and we have to updated it and upload it - about a minute
Joe Rafaniello
@jrafanie
Jun 14 2016 14:18
What about if we point at a tag or ref?
(git based)
Keenan Brock
@kbrock
Jun 14 2016 14:48
@jrafanie pretty sure we do a git fetch, which changes the local files - but unsure
I tried pointing it to a ref
and if the ref was local, I expected nothing to happen
but even pointing to a ref, when I bundle (locally) it hits github
I was trying to speed up bundle locally
(which means travis too)
Jason Frey
@Fryguy
Jun 14 2016 14:51
downloading a bundled gem or git based gems is faster than gem install / git clone respectively

if they are in parallel and there is a little duplicate work, you'll see multiple uploads of basically the same stuff

the cache stuff should not be paralled

unless you mean across suites, but even so, each suite is an enirely separate cache
Joe Rafaniello
@jrafanie
Jun 14 2016 14:53
Right. So, I don't know what to do about forks of gems if the intent is to remove git gems... do we rename ours so we can create real gems? do we try to become more involved in upstream so we can get our changes in and released?
Does it save us anytime if we do some of the git gems but not all of them?
Jason Frey
@Fryguy
Jun 14 2016 14:53
well the latter is always the best
I'm not sure about real gems
Keenan Brock
@kbrock
Jun 14 2016 14:53
real gems look locally and do nothing
partial conversion is better
Jason Frey
@Fryguy
Jun 14 2016 14:54
does the overhead in creating them and pushing them worth 41s?
Joe Rafaniello
@jrafanie
Jun 14 2016 14:54
Right, I don't know how much time we're saving to do some conversions to real gems
Keenan Brock
@kbrock
Jun 14 2016 14:54
every developer + every travis?
maybe not
Joe Rafaniello
@jrafanie
Jun 14 2016 14:55
Personally, I'd approach it more from a local setup
Keenan Brock
@kbrock
Jun 14 2016 14:55
we can fork the gems, or monkey patch them
Jason Frey
@Fryguy
Jun 14 2016 14:55
the gems are already forked
well most of them
Joe Rafaniello
@jrafanie
Jun 14 2016 14:55
are there things we can do to make bundle faster
Jason Frey
@Fryguy
Jun 14 2016 14:55
@jrafanie Not really no
it's not the bundle that's slow though
I mean bundler was 13s
Joe Rafaniello
@jrafanie
Jun 14 2016 14:56
Well, I know if we removed path gems, it would be faster
;-)
Jason Frey
@Fryguy
Jun 14 2016 14:56
a little, yeah :D
Joe Rafaniello
@jrafanie
Jun 14 2016 14:56
and less memory
Keenan Brock
@kbrock
Jun 14 2016 14:56
@jrafanie what did you just say?
Joe Rafaniello
@jrafanie
Jun 14 2016 14:56
0.5 second everywhere you load your bundle
Keenan Brock
@kbrock
Jun 14 2016 14:57
if we remove gems (in general)?
or remove gems that use git?
Joe Rafaniello
@jrafanie
Jun 14 2016 14:57
amazon/foreman gems using :path currently cause a re-resolve everytime you run bundle or require bundler/setup
Keenan Brock
@kbrock
Jun 14 2016 14:58
thanks
Joe Rafaniello
@jrafanie
Jun 14 2016 14:58
bundler/bundler#4618
I can't figure out the path feature in bundler to fix it
Keenan Brock
@kbrock
Jun 14 2016 14:58
was listening to rubyrogues over the weekend
the episode really encouraged using paths / lib before going over to real gems / separate repos
Jason Frey
@Fryguy
Jun 14 2016 14:59
paths on travis implies we'd need to vendor the gems, which I don't like
recap because I hate scrolling up:
section time notes
d/l cache 14.75s = 5.67s d/l + 9.08s add
bundle 13.46s with no updates to gems
u/l cache 41.81s
after_install 5.72
I didn't look into other bits of the test suite like creating the databases, cloning initial repo, etc
Keenan Brock
@kbrock
Jun 14 2016 15:02
yea, so it would be nice to avoid changing local files if there is no change
Joe Rafaniello
@jrafanie
Jun 14 2016 15:02
So, from a completely different perspective.. what about --fail-fast ?
Keenan Brock
@kbrock
Jun 14 2016 15:02
could people :+1: bundler/bundler#4618 ?
Joe Rafaniello
@jrafanie
Jun 14 2016 15:02
@kbrock it's not fixed
I can't figure it out
Keenan Brock
@kbrock
Jun 14 2016 15:02
ooh :(
Joe Rafaniello
@jrafanie
Jun 14 2016 15:03
it's WIP
Keenan Brock
@kbrock
Jun 14 2016 15:03
ugh
can you deps_for_source.sort != locked_deps_for_source.sort?
what is your problem?
lol: WHAT is YOUR problem?
Joe Rafaniello
@jrafanie
Jun 14 2016 15:04
I can't use their tests to recreate the exact problem because they're integration tests
Keenan Brock
@kbrock
Jun 14 2016 15:05
so this fixes the problem
but the slow bundle is sporadic
Joe Rafaniello
@jrafanie
Jun 14 2016 15:05
not really, I think there are different entry points to these methods
bundle check, vs. bundle, vs. require 'bundler/setup'
with/without Gemfile.lock, with/without changes to path gems
dependencies are hard
Keenan Brock
@kbrock
Jun 14 2016 15:06
yes
are there any non-integration tests?
Joe Rafaniello
@jrafanie
Jun 14 2016 15:07
Yes, but not in the code I want to change
It's 0.5 second and ~10 MB I think so I haven't really been trying hard
I think there's easier things to work on ;-)
Keenan Brock
@kbrock
Jun 14 2016 15:10
ok, so a) the code works, but the tests dont? Or b) the code only partially fixes the problem?
Joe Rafaniello
@jrafanie
Jun 14 2016 15:11
b)
Keenan Brock
@kbrock
Jun 14 2016 15:11
:shipit:
Joe Rafaniello
@jrafanie
Jun 14 2016 15:11
hehe
this is in the works though: CocoaPods/Molinillo#40
should land in bundler soon
Keenan Brock
@kbrock
Jun 14 2016 15:23
heh. +814/−199 -- big change