These are chat archives for sbt/sbt

5th
Apr 2017
Kai(luo) Wang
@kailuowang
Apr 05 00:27
I want to use a case class to represent a set of settings, I also want user to be able to set each single setting
What's the idiomatic way to achieve that?
Right now I have to repeat each setting name and type at three different places
First the setting key, then the default value for the key, then the case class instantiation
ritschwumm
@ritschwumm
Apr 05 00:57
make a trait/class with one instance per setting, where each instance contains the name, the default value and a function to put a value into your case class?
Dale Wijnand
@dwijnand
Apr 05 06:41
@adelbertc phew! Sanity restored.
Adelbert Chang
@adelbertc
Apr 05 06:41
@dwijnand indeed
thank you for dealing with my idiocy
Dale Wijnand
@dwijnand
Apr 05 06:44
no worries, it happens to everyone
@kailuowang could you provide an illustrative example?
having a Setting class made it easy to write internal methods, but tying it with all the setting keys values is a bit tedious. I was just curious if there is a better or at least more idiomatic way to achieve the same thing, i.e. tying multiple settings with an internal model.
Dale Wijnand
@dwijnand
Apr 05 13:30
I see. No, no way I see. All I can think is you can provide less.. So any change in setting requires calling copy on Settings. But if anything that's less idiomatic..
Kai(luo) Wang
@kailuowang
Apr 05 16:16
I see. Thanks @dwijnand
Ryan Williams
@ryan-williams
Apr 05 21:39

i heard that SBT's ivy-resolution behavior is configurable and can be made to mimic Maven's; is this true? Is there any documentation about this?

Furthermore, I heard (from @jodersky, iirc) that SBT and Maven's Ivy resolution differ in that SBT takes the version of an artifact that is latest across all occurrences in the transitive-dependency tree, while Maven takes the version that is closest to the top of the tree (i.e. if my project declares a dependency on Foo v1, that will be used even if some of my deps use Foo v2, since my project is the top of the dependency tree). Is that accurate?

OlegYch
@OlegYch
Apr 05 21:46
i don't think it is
afair maven would pick latest version amongst all transitive deps
Ryan Williams
@ryan-williams
Apr 05 21:48
mm i'm pretty sure that maven respects projects' declared version over deps' versions and SBT doesn't; let me try to test it :) pretty sure this was something that i stumbled on when switching from Maven to SBT
OlegYch
@OlegYch
Apr 05 21:49
maven has "dependencyManagement" thingy
which afair functions similar to dependencyOverrides in sbt
Ryan Williams
@ryan-williams
Apr 05 21:50

ah looks like this is the docs link i was missing

By default, the latest revision is selected.

OlegYch
@OlegYch
Apr 05 21:53
long
long story short all dependency management solutions for jvm have same shortcomings
Ryan Williams
@ryan-williams
Apr 05 21:55
well, sure, but that doesn't mean that the specifics of how SBT differs from Maven aren't worth better documenting. i hear Maven folks frequently use this as a reason for not switching to SBT
OlegYch
@OlegYch
Apr 05 21:55
probably because maven set the standard
Ryan Williams
@ryan-williams
Apr 05 21:55
sure, though that's orthogonal to what i'm after here
also there can be better and worse heuristics. in every case i've ever encountered i think "direct project dep" should be taken over "latest from the entire tree"
OlegYch
@OlegYch
Apr 05 21:56
well try it
i'd be surprised if whatever those people claim is true
Ryan Williams
@ryan-williams
Apr 05 21:57

working on a test maven project for it atm.

curiously, i don't see a conflict manager here that matches this understanding of what Maven does

do you think that Maven and SBT resolve version conflicts in the same way?
OlegYch
@OlegYch
Apr 05 21:57
yes
Ryan Williams
@ryan-williams
Apr 05 21:58
interesting. it is my impression that it is widely understood that they do not
OlegYch
@OlegYch
Apr 05 21:58
i've been using for 5 years and don't think it was any different
using maven
also 'selecting version from project instead of version required by dependency' is even more problematic than selecting newest version
because semver, you know
Ryan Williams
@ryan-williams
Apr 05 22:01
i am familiar with semver and don't see how your statement follows
OlegYch
@OlegYch
Apr 05 22:02
according to semver x.y.z+1 is strictly superior to x.y.z
Ryan Williams
@ryan-williams
Apr 05 22:07
not sure what "strictly superior" means here? i have seen many examples in the wild where taking a dep's newer version results in RTEs
OlegYch
@OlegYch
Apr 05 22:08
according to semver patch versions are fixing bugs
of course maven or whatever don't know about semver, so you might actually pull incompatible version
but it might as well be forward-incompatible
so either way stuff breaks
eg a lib requires guava 24, and you have guava 18 in your project
if you pick 18 the lib breaks
Ryan Williams
@ryan-williams
Apr 05 22:11
technically, no, it depends what code paths you hit at runtime
OlegYch
@OlegYch
Apr 05 22:11
if you pick 24 there is a good chance your project will work
Edmund Noble
@edmundnoble
Apr 05 22:11
@ryan-williams Semver requires that a new minor version cannot result in an RTE.
Otherwise they are doing semver incorrectly.
OlegYch
@OlegYch
Apr 05 22:11
especially since you can verify that
because tests and compiler, you know
which you typically can't run for the library
(to test if it still works with guava 18(
Ryan Williams
@ryan-williams
Apr 05 22:13
@edmundnoble today I had a project RTE against hadoop 2.6.0 but work on 2.7.3. do you think they are doing semver incorrectly?
Edmund Noble
@edmundnoble
Apr 05 22:13
No. That's not what @OlegYch is saying.
2.6.0 vs 2.6.1 should work.
I may be misspeaking; I suppose the last digit is not minor version.
Ryan Williams
@ryan-williams
Apr 05 22:14
yes, that is the "patch" version
Edmund Noble
@edmundnoble
Apr 05 22:14
Ah yes, just took a look.
Yeah, that's what Oleg is referring to above with "x.y.z+1" vs "x.y.z".
Ryan Williams
@ryan-williams
Apr 05 22:14
i understand
eugene yokota
@eed3si9n
Apr 05 22:14
some links on Maven vs sbt (Ivy) - sbt/sbt#2954
Maven uses nearest-wins algorithm on conflict
OlegYch
@OlegYch
Apr 05 22:15
  1. that 2.x is a minor version 2. you have rtes with downgraded version, which is exactly what i'm warning against
eugene yokota
@eed3si9n
Apr 05 22:15
Ivy uses latest-wins, but it encodes pom.xml with force(), which emulates nearest-wins
Ryan Williams
@ryan-williams
Apr 05 22:15
nearest-wins, thanks @eed3si9n, this is exactly what i was looking for
OlegYch
@OlegYch
Apr 05 22:16
isnt that specific to version ranges?
which are a big no-no
eugene yokota
@eed3si9n
Apr 05 22:16
it's to resolve conflict in general iiuc
OlegYch
@OlegYch
Apr 05 22:18
anyway, it would be interesting to see your sample project
utterly bizarre tbh
eugene yokota
@eed3si9n
Apr 05 22:29
yea. it would have been better if the notion of API compatibility was encoded into pom or ivy files
which leads to Adept, which leads to some tool that can extract JARs and treat them as sets of functions
OlegYch
@OlegYch
Apr 05 22:32
sounds nice
Ryan Williams
@ryan-williams
Apr 05 22:33
@eed3si9n what's Adept?
OlegYch
@OlegYch
Apr 05 22:33
i think unison attempts to do that
eugene yokota
@eed3si9n
Apr 05 22:34
Adept is the notion I mentioned at nescala of storing metadata of API compatibility in a Git-like thing on the side - https://www.slideshare.net/ekhfre/introducing-adept
http://adepthub.com/ says "Welcome to nginx!"
Ryan Williams
@ryan-williams
Apr 05 22:36
haha, cool, i remember you mentioning this now, thanks
Ryan Williams
@ryan-williams
Apr 05 22:48

an interesting wrinkle to semver-based heuristics: minor-version-upgrades cannot cause any new code paths in dependencies to be hit (without risking the same kinds of RTEs that major-version-upgrades can cause).

Suppose:

  • A:1.0.0 uses some library B:1.0.0
  • A:1.1.0 also uses B:1.0.0
  • A:1.1.0 exercises some code path X in B:1.0.0 that A:1.0.0 never uses
  • Library C uses A:1.0.0, and also B:2.0.0, which removed X.
  • C upgrades from A:1.0.0 to A:1.1.0, now gets RTEs (when its calls to A hit X) despite only having performed a minor-version upgrade.

semver is not obviously super clear on this from my reading:

MAJOR version when you make incompatible API changes,

that doesn't really seem to cover this scenario

MINOR version when you add functionality in a backwards-compatible manner

this could be interpreted as covering this scenario, but most ppl don't seem to interpret it that way, and the "MAJOR version" rule seems to miss it…

eugene yokota
@eed3si9n
Apr 05 22:51
B:1.0.0 and B:2.0.0 are incompatible so C can't use it
but if we do away with numbering system, we end up with "some tool that can extract JARs and treat them as sets of functions" that I mentioned above
Ryan Williams
@ryan-williams
Apr 05 22:53
agreed, i'm just chewing on what we could/can't get from some ideal semver-aware resolver :)
interesting point that a semver-aware resolver could at least flag the above scenario as invalid; however, that same resolver would flag the previous scenario, where A was happily depending on B:1.0.0, as invalid as well, which I'd call a false-positive
OlegYch
@OlegYch
Apr 05 22:54
sbt flags, at least in some cases
but semver is not nearly enough
Ryan Williams
@ryan-williams
Apr 05 22:55
and i would guess that something like 90-99% of production jvm deployments would be rejected for some conflicting major-version of something somewhere in the tree :) and yet if you don't get unlucky, you can get away with it
yes, agreed semver is a nice heuristic but i am interested in how we can do better, someday…
OlegYch
@OlegYch
Apr 05 22:59
build and test everything from source, that would be a step forward
from there you can start building various caching schemes
eugene yokota
@eed3si9n
Apr 05 23:00
that's what I told Ryan too, but he didn't like it
Ryan Williams
@ryan-williams
Apr 05 23:01
haha. afaict that is what https://leiningen.org/ does in clojure-land, which @danielcompton mentioned to me recently
OlegYch
@OlegYch
Apr 05 23:02
the important thing is too be able to turn caching off at any point
yeah, interpreted languages have it easier..
somewhat..
Ryan Williams
@ryan-williams
Apr 05 23:03

i mean, my dream-dep-manager will fall back to building from source against the deps that a given project needs, like the spark-notebook example i showed:

OlegYch
@OlegYch
Apr 05 23:03
not saying clojure is interpreted, just that it retained some properties
Ryan Williams
@ryan-williams
Apr 05 23:04
i think the real room for improvement is in analysis of the specific APIs that have changed, so that you can definitively say "this is fine" vs. "this is not fine" much more of the time

@OlegYch

the important thing is too be able to turn caching off at any point

why do you say that?

OlegYch
@OlegYch
Apr 05 23:06
to ease verification
and not end up with a bunch of binary blobs like we have now
speaking of which, is gentoo or other source based distros any better in this regard than regular linuxes?
Ryan Williams
@ryan-williams
Apr 05 23:13

@OlegYch probably superfluous at this point, but here's an example repo verifying nearest-wins vs latest-wins in maven vs sbt https://github.com/ryan-williams/maven-sbt-ivy-resolution

maybe at the least it will add some pagerank points to eugene's links above :)

OlegYch
@OlegYch
Apr 05 23:13
cheers
Ryan Williams
@ryan-williams
Apr 05 23:16

however, i guess i still have my original issue: i heard that SBT could be configured to use Maven-style resolution.

I see that it can use the ivy conflict mgrs here: https://github.com/sbt/sbt/blob/v0.13.13/ivy/src/main/scala/sbt/IvyInterface.scala#L43-L50

but i don't see Maven's "nearest-wins" anywhere!

OlegYch
@OlegYch
Apr 05 23:16
good
perhaps coursier has something similar?
OlegYch
@OlegYch
Apr 05 23:33
tbh it's the first time i hear about conflict managers other than default and strict, and strict is not useable
Ryan Williams
@ryan-williams
Apr 05 23:36

poking around, i don't see coursier obviously supporting "nearest wins".

i think i agree with you that strict is not usable and that version ranges and semver can't really save us.

i don't think the "nearest wins" heuristic is so unreasonable though, and seems like a fine complement to the "latest wins" heuristic; SBT's dependencyOverrides hooks seem to validate the use-case, and i've only recently become aware of them, so it's possible i'll be able to use them to get out of tricky situations in the future