These are chat archives for sbt/sbt

13th
Jun 2017
eugene yokota
@eed3si9n
Jun 13 03:40
back published a bunch of plugins for sbt 1.0.0-M6 - http://www.scala-sbt.org/1.0/docs/Community-Plugins.html
N.S. Cutler
@godenji
Jun 13 06:54
trying to sort out why test dependencies are not recognized in a scala.js cross project build)
duplicating foo % "compile -> compile; test -> test" for each cross built project brings in the test dependencies but I'd prefer to define the dependency right at the source and be done with it.
if it helps the target project I'm depending on is a ProjectRef.
N.S. Cutler
@godenji
Jun 13 07:56
nevermind, sorted out on scala.js gitter channel.
Anton Sviridov
@keynmol
Jun 13 09:53

Hi all. Is there a way to mark dependency as provided for certain stages? I.e. I have something like this:

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.2" // % "provided"
...
def runtimeSparkDependency: Seq[ModuleID] => Seq[ModuleID] = { _.collect{
  case dependency if dependency.name == "spark-core" =>
    println(dependency)
    "org.apache.spark" %% "spark-core" % "1.6.2" % "provided"
}}

libraryDependencies in assembly ~= runtimeSparkDependency

But this doesn't seem to work and the spark-core jar is still packaged

Basically trying to make sbt run work as well as submitting the fat jar to spark-submit
Dale Wijnand
@dwijnand
Jun 13 09:59
:+1:
Anton Sviridov
@keynmol
Jun 13 09:59
oh nice, lemme try that
Smashing! thanks @jeffreyolchovy
Jeffrey Olchovy
@jeffreyolchovy
Jun 13 10:11
@adelbertc did you solve your issue wrt release and compileOrder? given a trivial project with one Java class and one Scala object where the Scala object depends on the Java class, a release process that compiles and runs tests using the built-in release process works without issue, respected the declared compileOrder. what does your releaseProcess look like? is it custom or non-standard in anyway?
Mirco Dotta
@dotta
Jun 13 10:47
I’m implementing a task in a AutoPlugin, and I’m wondering if it is possible within its implementation to execute another project’s task without running the latter on the project’s aggregates as well.
Otherwise said, can I programmatically disable aggregation when evaluating a project’s task?
Dale Wijnand
@dwijnand
Jun 13 10:51
I think you might, by going out of the task graph - which is not recommended
Mirco Dotta
@dotta
Jun 13 10:52
Hey @dwijnand , how can I achieve that? (I understand this is not the recommended way, but I’m attempting to time execution of the compile task, and hence I need to control what’s being compiled)
By "how can I achieve that” I mean: how do I get “out of the task graph”?
any resource/link/hint?
Dale Wijnand
@dwijnand
Jun 13 10:54
Extracted#runTask (as opposed to runAggregated)
hush hush :P
Mirco Dotta
@dotta
Jun 13 10:55
ahah, thx!
Dale Wijnand
@dwijnand
Jun 13 10:55
here's an example of why it's a bad idea: sbt/sbt#2970
Mirco Dotta
@dotta
Jun 13 10:58
mmmm, I’ll keep that in mind - but I don’t really see an alternative if I want to obtain the compile time of single projects, without their aggregates
Dale Wijnand
@dwijnand
Jun 13 10:59
generally I've seen aggregating projects not have any code
Like empty roots (that also shouldn't publish empty jars)
Mirco Dotta
@dotta
Jun 13 11:00
that’s true - thanks for hinting a pragmatic approach may be wiser :)
Jorge
@jvican
Jun 13 11:40
Given the fact that you want to disable aggregation, you may be lucky if you use aggregate in yourTask := false
Definition of aggregate: val aggregate = SettingKey[Boolean]("aggregate", "Configures task aggregation.", BMinusSetting).
Otherwise, Dale's suggestion is good, I use runTask all over the place, and other popular plugins like sbt-release do too.
Mirco Dotta
@dotta
Jun 13 11:43
@jvican Hey Jorge! I knew about aggregate in yourTask := false, but since aggregate is a setting, I was assuming I can’t just turn it off and on on-demand (i.e., in the implementation of my task). Btw, thanks to the pointer to sbt-release for runTask usage
Jorge
@jvican
Jun 13 11:46
Yw! Just make sure that the things you run with runTask do not contend for resources (as update does).
Mirco Dotta
@dotta
Jun 13 11:47
:+1:
Jorge
@jvican
Jun 13 11:47
Another idea sprang to mind: you can create a synthetic project per project that contains your task. You define the synthetic project without aggregated projects (or disabling aggregation for that project), and whenever you want to run yourTask you just execute it in the mapped project.
You can get the map from originalProject to syntheticProject either by defining a setting in every project that points to the synthetic project, or by getting all the projects from the state.
This solution is fancier, and I must admit I've never used it. But since you don't use runTask, you ensure yourself that you won't have any race condition.
Mirco Dotta
@dotta
Jun 13 11:50
Yep, sounds good. Thanks a ton for all the ideas!
Jorge
@jvican
Jun 13 11:51
:+1: :)
Sébastien Doeraene
@sjrd
Jun 13 14:40
I was trying to get the TaskStreams (what streams.value gives you) of an outer task from within an anonymous Def.task. (I ended up finding a completely different solution, but I'd still want to get to the bottom of this issue. So this for "academic" purposes only.) I managed to accomplish it using the changes visible in the following branch: https://github.com/scala-js/scala-js/compare/0.6.x...sjrd:no-taskdyn-wip?w=1#diff-dd593403434ec9de3670cb5722f10171
For convenience, I repeat the gist here:
      key := Def.settingDyn {
        val thisP = thisProjectRef.value
        val config = configuration.value
        Def.task {
          val st = state.value
          val buildStruct = buildStructure.value
          EvaluateTask.withStreams(buildStruct, st) { streams =>
            val s = EvaluateTask.getStreams(key in Global in (thisP, config), streams)

            //val s = streams.value
            val log = s.log
            ...
        }
      }
This is extremely ugly. Not the least ugly part is that I need to access state.value and buildStructure.value from the task, which is an abomination.
Is there a clean way to do this?
(Note that the "obvious" solution of putting val s = streams.value outside of the anonymous task doesn't work, because it's a settingDyn outside, and a setting cannot depend on the task streams.)
(The other "obvious" solution to replace the settingDyn by a taskDyn means that it can never benefit from sbt/sbt#3258 .)
Dale Wijnand
@dwijnand
Jun 13 14:54
@sbtUnchecked?
Justin Kaeser
@jastice
Jun 13 14:58
any good alternatives to supplying values to a task from a caller without making it an inputTask?
Dale Wijnand
@dwijnand
Jun 13 14:58
parsing /dev/urandom? :^)
oh, right, good alternatives
Justin Kaeser
@jastice
Jun 13 14:59
:trollface:
parameters, actually
I could write it to a file and parse that from a task, but that's just messy
Dale Wijnand
@dwijnand
Jun 13 14:59
make the task return a function and then call the function?
Justin Kaeser
@jastice
Jun 13 15:00
huh, I'll try if that fits
but
how will I have the caller provide the parameters
I basically want an inputTask but with optional input so that non-input tasks can depend on it
depending on whether there is input? if that makes sense
Dale Wijnand
@dwijnand
Jun 13 15:06
non-input tasks can depend on input tasks though, no? By hand-passing in no input?
toTask iirc
mm12333
@mm12333
Jun 13 15:08

hi all.
i have a multi module project, and i have a version definition like: version in ThisBuild := 1.0.0
then i run sbt cli, and set version like: set version := 2.0.0
i check version:

version

output:

project1:/*version
1.0.0
project2:/*version
1.0.0
root:/*version
2.0.0

How to redefine version for all projects instead of root only?

Sébastien Doeraene
@sjrd
Jun 13 15:10
@dwijnand What's @sbtUnchecked?
Dale Wijnand
@dwijnand
Jun 13 15:11
Maybe I missunderstood your question. Are you avoiding the new linting errors/warnings?
I assumed that's why you weren't using val s = streams.value inside Def.task
Sébastien Doeraene
@sjrd
Jun 13 15:12
No it's in stock 0.13.15.
I avoid your suggestion because otherwise I get the same generic TaskStreams for global/global/global, and that means cacheDirectory and last are broken.
Dale Wijnand
@dwijnand
Jun 13 15:14
TIL, I see.
Justin Kaeser
@jastice
Jun 13 15:17
yes. but if that gives them a fixed input. my problem is kind of I want some options to be either input (when provided) or default otherwise, so that it's transparent to the tasks depending on these parameters
Dale Wijnand
@dwijnand
Jun 13 15:21
I don't get it.
Justin Kaeser
@jastice
Jun 13 15:25
say I have a task params that provides some parameters to other tasks. If I make it an inputTask, all tasks that depend on params will need to be inputTasks too if they want to access the parameters that user provided when calling
Dale Wijnand
@dwijnand
Jun 13 15:31
Ah, I see now. Yeah..
Justin Kaeser
@jastice
Jun 13 15:33
so since I'm doing this programatically, I could pass it as a file or environment or whatever. just kind of messy
Dale Wijnand
@dwijnand
Jun 13 15:45
Sounds like you want to make everything an input task.
So it could take and interpret user input.
Justin Kaeser
@jastice
Jun 13 15:56
hmmmaybe
ok I'll try it
Justin Kaeser
@jastice
Jun 13 16:07
but it looks like then some task or other might capture the input and the other tasks get a different value
Jorge
@jvican
Jun 13 18:03
@sjrd Context: i was discussing that ugly piece of code with Seb over lunch and i agree we have to get to the bottom of that issue.
How do you think we could improve sbt to provide you streams.value for the wrapping task?
An easy solution would be to create a task that exposes that funcionality, with a similar implementation to the one you've provided.
In any case, before proceeding to discuss this issue, @sjrd are you sure that implementation works? what happens if the logs already exist? You're doing withStreams and that reads as if you're creating new streams for a task.
Sébastien Doeraene
@sjrd
Jun 13 18:08
I haven't done extensive testing other than for my use case. In my case it works because the outer task doesn't do any logging of its own.
I'm pretty sure that if the outer class also does logging, my implementation will erase the logging previously done by the outer task when the inner task starts, so last won't show stuff coming from the outer task.
Jorge
@jvican
Jun 13 18:09
Exactly, that's my intuition too.
Sébastien Doeraene
@sjrd
Jun 13 18:10
My ideal UX for this issue would be that streams.value in an anonymous Def.task returned by some non-anonymous dynamic task returns the same value as streams.value for the enclosing task, and fuses everything together nicely.
But I have no idea how difficult it would be to achieve that. It might be structurally impossible.
I won't fight for this. I found a (better) totally different solution to the problem I was tackling in the first place.
Someone else could fall in the same situation one day, but it's pretty unlikely compared to everything else that could happen.
I would put this issue very low on the priority list.
Jorge
@jvican
Jun 13 18:14
Can you file an issue in sbt/sbt? That way we won't miss it in the future. I think it's important that you document it just in case someone stumbles upon it again.
Sébastien Doeraene
@sjrd
Jun 13 18:17
Yeah maybe I'll do that when I'm motivated enough to do so.
Jorge
@jvican
Jun 13 18:18
:+1:
Thanks.
Sébastien Doeraene
@sjrd
Jun 13 18:18
But I'm not sure it's good for the posterity that my solution be visible to anyone would be trigger-happy to apply it. It's an evil workaround, not a solution. I mentioned it more as an executable spec of what I wanted to achieve in a clean way.
Jorge
@jvican
Jun 13 18:19
At least it's something. In the future, I can look into it and try to find a better solution. I'm just proposing it because I know that if someday I choose to do so, I won't be able to find this chat. :)
Adelbert Chang
@adelbertc
Jun 13 19:55
what is the ivy pattern to get say, org/typelevel versus org.typelevel? it seems theres just [organization]
OlegYch
@OlegYch
Jun 13 20:05
i think you need to use maven compatible pattern
sbt.Patterns#isMavenCompatible
Adelbert Chang
@adelbertc
Jun 13 20:12
m2compatible
im doing it in the context of repo.properties though
Sukant Hajra
@shajra
Jun 13 20:51
I'm looking at the source code for WartRemover, and I'm a touch confused by something.
Wartremover sets itself as "projectSettings". But some of the switches for scalacOptions are set with "inScope(This)"
And I'm kind of confused by that.
Is "This" the same as the "this build" scope? And if so, that seems in tension with the fact the setting is put in the "projectSettings" field.
Jorge
@jvican
Jun 13 20:53
val ThisScope: Scope = Scope(This, This, This, This)
case object This extends ScopeAxis[Nothing]
Sukant Hajra
@shajra
Jun 13 20:53
I guess the summarizing question is, what is the semantic of this? override projectSettings = Seq(inScope(This) { ... })
Oh, I should better understand "ThisScope"
Jorge
@jvican
Jun 13 20:54
/**
 * This is a scope component that represents not being
 * scoped by the user, which later could be further scoped automatically
 * by sbt.
 */
Sukant Hajra
@shajra
Jun 13 20:55
that's kind of complicated, I feel.
all SBT code is "by the user" when it comes to plugins.
Jorge
@jvican
Jun 13 20:55
no, when it comes from plugins it's not "by the user"
Adelbert Chang
@adelbertc
Jun 13 20:56
ah there's a , mavenCompatible you can set in repo.properties http://www.scala-sbt.org/0.13/docs/Launcher-Configuration.html#Syntax
Sukant Hajra
@shajra
Jun 13 20:57
@jvican that's fine. but I still need to understand how it gets set. I understood the simplier world where settings were explicit about their scope.
I just need an algorithm I can keep in my head. Otherwise, I'm just deeper in SBT code, which is... well... kind of a time-sink.
Sébastien Doeraene
@sjrd
Jun 13 20:58
@shajra This is a marker that, when the setting is passed through some filters, will be replaced by an appropriate value.
Jorge
@jvican
Jun 13 20:58
Don't worry, this is advanced stuff. This is a scope component. To make an anology, it's the equivalent of None of Option
Sébastien Doeraene
@sjrd
Jun 13 20:58
For example, all the settings in a project's .settings(...) call will be "filtered through" that project as an axis.
This means that all the Thises in the project axis will be replaced by the project on which you call .settings
Sukant Hajra
@shajra
Jun 13 20:59
I see.
Sébastien Doeraene
@sjrd
Jun 13 21:00
If you write an sbt plugin, and you want some settings to be different per configuration, you would leave out the Config axis as This,, build a large Seq[Setting[_]] that need to be config-specific (usually called configSettings), and then have inConfig(Compile)(configSettings) ++ inConfig(Test)(configSettings).
Jorge
@jvican
Jun 13 21:00
For instance, if you do compile in Test then internally that's rewritten as in(This, Select(Compile), This)
Sébastien Doeraene
@sjrd
Jun 13 21:00
The inConfig(Compile) explicitly passes the configSettings through a "filter" that replaces all the Thises in the Config axis by Compile.
Sukant Hajra
@shajra
Jun 13 21:02
@sjrd that makes it seem like all settings are intrinsically scoped as This. Is inScope(This) redundant?
Sébastien Doeraene
@sjrd
Jun 13 21:02
And, no, This has nothing to do with ThisBuild.
inScope(This) is definitely redundant.
It would replace all the Thises by ... Thises :p (I guess, I've never tried it)
Sukant Hajra
@shajra
Jun 13 21:04
@sjrd okay, so all this This stuff is kind of new to me... but I thought the same thing is accomplished by the delegation chain of scopes.
Jorge
@jvican
Jun 13 21:04
@sjrd It seems we have a different understanding of what This is.
Sukant Hajra
@shajra
Jun 13 21:04
I don't know who to trust between the two of you.
Jorge
@jvican
Jun 13 21:04
Just hold on.
shajra @shajra holding on.
Sébastien Doeraene
@sjrd
Jun 13 21:04
@jvican You see it from the point where you write the sbt "syntax" with in until it forms a Scope. I am explaining what that Scope becomes afterwards.
The two things are true, and complementary.
Jorge
@jvican
Jun 13 21:05
@sjrd Right, that's what I'm trying to figure out.
I misunderstood your comments.
Sébastien Doeraene
@sjrd
Jun 13 21:06
@shajra The delegation chain of scopes is different. For example scope delegation is not sufficient to replicate settings in Compile and in Test, allowing one to change one but not the other.
Sukant Hajra
@shajra
Jun 13 21:06
@sjrd @jvican okay so we're in agreement that this wrapping of inScope(This) is redundant? https://github.com/wartremover/wartremover/blob/master/sbt-plugin/src/main/scala/wartremover/package.scala#L18
Jorge
@jvican
Jun 13 21:07
I'm gonna try to summarize @shajra: This is like an empty variable in a coordinate; (x, 3, y) would be in(This, Select(FixedConfiguration), This). It's like saying: "Hey, sbt, put whichever value you're supposed to put here, I don't care.".
Sébastien Doeraene
@sjrd
Jun 13 21:07
@shajra Yes, unless I am contradicted by someone even better informed about the internals of sbt than I am.
Sukant Hajra
@shajra
Jun 13 21:09
Is This only for the configuration axis?
where ThisScope is all the axes?
Jorge
@jvican
Jun 13 21:10
No, you use it for all the axes.
in(This, This, This)
Sukant Hajra
@shajra
Jun 13 21:10
Okay, then if This is for all axes, then I don't know which axis we're talking about with inScope(This).
oh, probably whatever the first position is.
no, I'm confused. in the source, inScope take a type of "Scope". So This seems to work for the entire scope... all axes, I guess?
Jorge
@jvican
Jun 13 21:14
inScope(This) will put This to every axis, yes
Sukant Hajra
@shajra
Jun 13 21:15
Okay, so let me back up to my original question, since it seems that the This part of the code might not be related.
I see a compiler plugin added.
which I believe adjusts the key compile:scalacOptions for the project.
But the scalacOptions right after the addition of the compiler plugin are added for the global configuration, right?
so with delegation, I'm not sure how they'd be registered at all.
(but I'm pretty sure WartRemover works).
Does my question make sense?
Jorge
@jvican
Jun 13 21:22
So, I've been quiet all the time because there was something that didn't make sense. How is This being accepted by in if This is a ScopeAxis and in expects a Scope?
It happens that sbt 0.13 has an implicit conversion from ScopeAxis to Scope: main/settings/src/main/scala/sbt/ScopeAxis.scala 22: implicit def scopeAxisToScope(axis: ScopeAxis[Nothing]): Scope
It took me a while to find it because this has disappeared in sbt 1.0 (thanks God).
Sukant Hajra
@shajra
Jun 13 21:24
wait a second, I'm being silly. addCompilerPlugin only adds to the libraryDependencies, not scalacOptions.
Jorge
@jvican
Jun 13 21:24
So, in essence, i confirm what i said before, inScope(This) scopes a key for all This.
Sukant Hajra
@shajra
Jun 13 21:25
@jvican so are you arguing the inScope(This) does indeed make a difference?
Jorge
@jvican
Jun 13 21:25
I don't think inScope(This) is redundant @sjrd @shajra.
Yes, indeed.
Sukant Hajra
@shajra
Jun 13 21:26
Okay, so this might make sense for why I'm seeing compile:scalacOptions set, even though I can't find that explicitly set anywhere?
Jorge
@jvican
Jun 13 21:27
Yes. derive is scoping it for all the scopes, removing any project constrain.
Pretty magic, didn't know about that use case.
Sukant Hajra
@shajra
Jun 13 21:28
oh yeah, derive... I hadn't gotten to that part yet.
Jorge
@jvican
Jun 13 21:29
Wow, you're real tenacious ;)
So what derive is doing is to define a key in whichever scopes the static dependencies of that task are defined.
Sukant Hajra
@shajra
Jun 13 21:33
@jvican can you unpack that a little? I'm not sure I follow a sentence that terse.
Jorge
@jvican
Jun 13 21:33
scalacOptions is not a good example for this...
In this case, derive is redefining scalacOptions in all the scopes it's already defined.
Sukant Hajra
@shajra
Jun 13 21:33
I might have found a good example of this: https://gist.github.com/eed3si9n/bf561bce36ca72b5f659
Jorge
@jvican
Jun 13 21:34
But to show the power of derive, I'll illustrate it with a more complicated example.
Imagine you have a setting a and it's only defined in Compile.
You want to define a task b that depends on a, but you don't know where a may or may not be defined (i.e. in which scope it is)
Sukant Hajra
@shajra
Jun 13 21:35
@jvican can you check the gist I just posted? I wouldn't want to put you out explaining it.
Jorge
@jvican
Jun 13 21:35
Yeah, exactly, that explains it :) I won't continue my explanation.
Adelbert Chang
@adelbertc
Jun 13 22:10
is it possible to configure JVM options in the build.sbt of a project, for SBT? e.g. instead of running sbt -Divy.home=<path> -Dsbt.override.build.repos=true configuring build.sbt in such a way so i can just run sbt instead?
Jorge
@jvican
Jun 13 22:13
Not that I know of :|
Oh well, it seems this is what you're looking for: SBT_OPTS environment variable holding either the sbt args directly, or the reference to a file containing sbt args if given path is prepended by '@' (e.g. '@/etc/sbtopts') Note: "@"-file is overridden by local '.sbtopts' or '-sbt-opts' argument. -sbt-opts <path> file containing sbt args (if not given, .sbtopts in project root is used if present)
It's in sbt-extras, though. Don't know if that's possible with the normal launcher.
Adelbert Chang
@adelbertc
Jun 13 22:29
hmmm alright
thanks