These are chat archives for sbt/sbt

29th
Mar 2015
Sam Halliday
@fommil
Mar 29 2015 10:09
has something happened to the org.scala-sbt#sbt-maven-resolver;0.13.8!sbt-maven-resolver.jar artefacts? I'm consistently failing to download them in our CI jobs
try building this docker image for confirmation https://github.com/ensime/ensime-docker/tree/master
aaah! Never mind, I think I know what it is. I need the mavenResolverPlugin in one of the cache generator branches
hmm, actually, it should already be there. Nope still broken.
Mateusz Czeladka
@matiwinnetou
Mar 29 2015 12:49
I am dying to see SBT in Action in a paper form soon, any ETA on release of print version of the book?
Dale Wijnand
@dwijnand
Mar 29 2015 13:35

@matiwinnetou: from a couple of days ago:

@dwijnand when was SBT in Action due to be out again?
@jsuereth I dunno. Copyediting is only done w/ chapter 5. It's been like 1 month a chapter for reviews

Mateusz Czeladka
@matiwinnetou
Mar 29 2015 13:37
Thx, actually Josh mentioned once in a video that SBT doesn’t have a name, it turns out amazon calls it: "SBT in Action: The Simple Scala Build Tool”, so this no name to SBT didn’t really work out.
Dale Wijnand
@dwijnand
Mar 29 2015 13:40
Hehe well it seems like it's a generally avoided question. To me it was originally "simple build tool", then it started getting mocked for "simple" and transitioned to "Scala Build Tool"
Mateusz Czeladka
@matiwinnetou
Mar 29 2015 13:40
with .value macros it is actually not that hard anymore if one stays away from things like commands
Dale Wijnand
@dwijnand
Mar 29 2015 13:43
It's definitely complicated, as over time it explored how to solve the domain, but now with that experience it's hard to re-simplify because it's (correctly imo) giving more importance to maintaining binary compatibility for plugins. I expect whatever version will finally break that compatibility can thrive in a simplified architecture.
Josh Suereth
@jsuereth
Mar 29 2015 16:16
Simple = a concept once learned applies everywhere. Similar to how scala is simpler than java (viewed from the right angle). Turns out not everyone agrees this is what simple means.
Additionally our APIs on top of core (like incremental compiler, logging, actually writing commands) are not simple in the same way Tasks and Settings were meant to be.
If we could break something, I'd try to find way to have plugins/projects all on separate clasloaders, or lazily loaded
Right now the model forces everything to be seen/loaded
And that's one of the biggest complaints currently (see sbt-dev)
Sam Halliday
@fommil
Mar 29 2015 20:23
@jsuereth you must be kidding. That isn't even in the top 10. Time to update the dependency list has got to be the biggest complaint, said every user of sbt ever.
the biggest fundamental usability problem facing sbt is the time to perform ivy resolution. Then it's the API, which is massively overcomplicated. But I'm willing to live with the legacy of the API because it's a legacy that we've all bought into because there is no mature alternative. The sbt updateClassifiers command (used by IntelliJ 14 and anyone using sbt directly) is slower than the compile even on gigantic projects, and is the bottleneck in all CI jobs, even with jars pre-downloaded.
Sam Halliday
@fommil
Mar 29 2015 20:28
If you really believe that lazy loading of plugins / classloader magic is even close to ivy resolution, in terms of user pain, I strongly recommend you do a user survey.
Dale Wijnand
@dwijnand
Mar 29 2015 20:30
Personally I don't suffer from update time, because 1 don't change dependencies much and 2 I have much bigger delays in CI, specifically a lack of a near proxy, just my 2c
Sam Halliday
@fommil
Mar 29 2015 20:30
@dwijnand you probably don't notice it because you don't ever clean your tree
ever since Eugene's caching, things have gotten a lot faster. Plus, how big is your typical project?
Dale Wijnand
@dwijnand
Mar 29 2015 20:31
what does clean my tree mean?
Sam Halliday
@fommil
Mar 29 2015 20:31
git clean -xfd
Dale Wijnand
@dwijnand
Mar 29 2015 20:31
not git clean, but I sbt clean
Miles Sabin
@milessabin
Mar 29 2015 20:31
@fommil did you just send @dwijnand to his room to tidy up?
Sam Halliday
@fommil
Mar 29 2015 20:31
hehe
Dale Wijnand
@dwijnand
Mar 29 2015 20:31
:p
Sam Halliday
@fommil
Mar 29 2015 20:32
sbt clean doesn't erase its cache of the dep resolution
Dale Wijnand
@dwijnand
Mar 29 2015 20:32
my typically project is 1-3 modules
"well thars your problem" :P
Sam Halliday
@fommil
Mar 29 2015 20:33
FYI, in my project freeslick, it takes longer to do the sbt update than it does to start up MSSQL 2000, 2005, 2008 in virtual machines, compile the project, and run the tests. And that's with all jars pre-downloaded.
it's similar in ENSIME, where it takes longer to do the update than to compile everything... and that's using shapeless! @milessabin :-P
Miles Sabin
@milessabin
Mar 29 2015 20:34
Wow!
Sam Halliday
@fommil
Mar 29 2015 20:34
you should see how much docker magic I have to do in our ENSIME CI setup to try and get it as low as 2 minutes
Josh Suereth
@jsuereth
Mar 29 2015 20:34
@fommil I'm being optimistic that cached resolution cleans that complaint up. You're right it has been #1 but we're actively fixing that.
Sam Halliday
@fommil
Mar 29 2015 20:35
@jsuereth nope, caching is a help but it's miles and miles from a real fix
Josh Suereth
@jsuereth
Mar 29 2015 20:35
The next complaint (which could cause shift) is multi project builds.
Sam Halliday
@fommil
Mar 29 2015 20:35
the gap is a lightyear
also, caching only helps repeated invocations... it still kills project setup times and CI jobs
Dale Wijnand
@dwijnand
Mar 29 2015 20:36
what's the complaint around multi project builds?
Josh Suereth
@jsuereth
Mar 29 2015 20:36
It should also help for inter project dependencies, unless we don't see enough cache hits
Dale Wijnand
@dwijnand
Mar 29 2015 20:36
aah, I see
Sam Halliday
@fommil
Mar 29 2015 20:38
I'm really hopeful that Eugene is able to get updateClassifiers working fast (it doesn't currently), which is going to be really awesome. But it still doesn't fix the CI problem.
Josh Suereth
@jsuereth
Mar 29 2015 20:38
I think update classifiers is aping maven, which is pretty wrong for ivy
Sam Halliday
@fommil
Mar 29 2015 20:38
and, on big projects, the "run sbt update on a clean checkout" problem
Josh Suereth
@jsuereth
Mar 29 2015 20:39
I.e. its architectef slow
Sam Halliday
@fommil
Mar 29 2015 20:39
btw, I have enabled the maven plugin in my projects but it doesn't seem to make any difference. Should I have noticed a speedup, or even use of the ~/.m2 ?
everything is still being saved to my ~/.ivy2/cache
Josh Suereth
@jsuereth
Mar 29 2015 20:40
The first part of that is aimed at correctness. It'll still be ivy slow for a variety of reasons
Sam Halliday
@fommil
Mar 29 2015 20:40
mmm, maven is super fast in java projects. I don't see why its use as a library should be so much slower. Is it being forced into single threaded mode or something?
Josh Suereth
@jsuereth
Mar 29 2015 20:40
One is the ivy http layer which we hope to fix
Sam Halliday
@fommil
Mar 29 2015 20:41
hmm, actually I disagree with HTTP bit being slow. That is a one-off thing that can be solved with userland caching.
The real bottleneck is doing the local resolution.
I'm willing to be tolerant of slow downloads, plus I have automated docker jobs to solve that problem for me.
Havoc Pennington
@havocp
Mar 29 2015 20:41
I still think the solution there is to make resolution a maintainer-only operation. CI should get a list of urls with sha hash, run N stat to check the cache, then run N downloads in parallel and be done. No computation at all; no downloads that have to wait for other downloads at all. But, along with every other JDK developer for the last 15 years I haven't coded it even though there's nothing that should be that hard. But I would love to sometime.
Sam Halliday
@fommil
Mar 29 2015 20:42
the bottleneck is not the downloading
the bottleneck is sbt update after everything has been downloaded already
Havoc Pennington
@havocp
Mar 29 2015 20:42
should download OR resolve. for a reproducible build you don't want to compute anything. I don't anyway.
shouldn't
still have a 2 minute sbt update stage
i.e. even when using the docker image that pops out. That's a purely local sbt update problem.
same thing on all my projects, same thing I see on the desktop of every developer I work with
Havoc Pennington
@havocp
Mar 29 2015 20:45
I don't know what that script does or why update is slow for you. I haven't dug into this. I'm just saying conceptually all the work can be pre-done and checked into git.
but literally tens of thousands of devs have been annoyed by this without fixing it so I'm not very original
Sam Halliday
@fommil
Mar 29 2015 20:47
the script does sbt updateClassifiers on all my branches and then caches the results, so that the next build (including PRs) reap the benefits
so, it basically does what you suggested: downloading becomes a maintainer-only problem :-)
I've often thought about rewriting the Ivy resolver, but I suspect the integration work with SBT is the real problem, not coming up with a fast graph resolution algorithm.
Havoc Pennington
@havocp
Mar 29 2015 20:57
sounds like running a profiler would tell something
or just strace. are you sure it isn't reading a boatload of xml files and crap?
eugene yokota
@eed3si9n
Mar 29 2015 23:57
@fommil there was similar feature request around assembly's zip caching. changing the cache directory shouldn't be too hard, but you then would need something like reallyCleanForRealz task if we implemented update that survives clean (in a way cached resolution is an attempt at that)