These are chat archives for oncue/knobs

27th
Apr 2016
Gastón Tonietti
@ktonga
Apr 27 2016 00:40
hey @timperrett hope you're doing well. Sorry to bother you. Were you able to take a look at the PRs? They are small and just trivial changes, should be quick to review them. Cheers!
Timothy Perrett
@timperrett
Apr 27 2016 04:43
@runarorama @stew i’m making scalaz-7.1 and scalaz-7.2 branches on both ermine parser and knobs to accomodate this; any suggestions on how we should name release artifacts? Options are: 1. change the group or artifact IDs. 2. Change the version with a suffix or classifier 3. Only publish from one branch
whatever teh case, we need to figure out how exactly we are going to handle the releases
Daniel Spiewak
@djspiewak
Apr 27 2016 04:44
@timperrett given that SBT sits on ivy, isn't there a random metadata feature we can stuff things like this into?
I vaguely remember Mark Harrah telling me about something like that
it was the whole reason he couldn't move SBT over to Aether
Timothy Perrett
@timperrett
Apr 27 2016 04:45
@djspiewak yeah there is that, but i’d rather use a classifer so its maven compatible, as we have no idea what tools people out there might be using
people still use maven, believe it or not
Daniel Spiewak
@djspiewak
Apr 27 2016 04:45
true, though what are the odds that they're using maven with knobs
Timothy Perrett
@timperrett
Apr 27 2016 04:46
@djspiewak im pretty sure @tpolecat uses maven
so its not that uncommon :P
Daniel Spiewak
@djspiewak
Apr 27 2016 04:46
Timothy Perrett
@timperrett
Apr 27 2016 04:46
but yeah, point taken, we have options
@djspiewak what do you think about the branches? it was the best thing i could think of, but i dont really like it
Daniel Spiewak
@djspiewak
Apr 27 2016 04:47
I really really really don't like it
I think you'd be better off having source directories
Timothy Perrett
@timperrett
Apr 27 2016 04:47
yeah
same
@djspiewak errrr
Daniel Spiewak
@djspiewak
Apr 27 2016 04:47
well, the idea is basically the same as the compat directories thing for cross building
Timothy Perrett
@timperrett
Apr 27 2016 04:47
and how exactly would that happen? i mean, that would be a shit ton of sbt trickery to get it releasing right
Daniel Spiewak
@djspiewak
Apr 27 2016 04:47
you know how src/main/scala isn't actually the only dir on the path right?
src/main/scala-2.11 is also on there
Timothy Perrett
@timperrett
Apr 27 2016 04:47
of course :)
Daniel Spiewak
@djspiewak
Apr 27 2016 04:47
or scala-2.10
so the only trickery is surrounding the lifecycle
I can think of two ways to do it
first, we can sort of double-fake the cross build infrastructure and make it think it has two separate cross builds for the same scala version
as long as we can identify the difference, we can have different dependencies
Timothy Perrett
@timperrett
Apr 27 2016 04:48
@ktonga just in case you’re not following, we may want to do something different :)
@ktonga so hold your horses with the PR
Daniel Spiewak
@djspiewak
Apr 27 2016 04:49
the second is that we jury rig some cross build infrastructure by fiddling with the lifecycles
Gastón Tonietti
@ktonga
Apr 27 2016 04:49
ok give me a sec for catching up
Timothy Perrett
@timperrett
Apr 27 2016 04:49
@djspiewak remember we’re using the travis build matrix,
Daniel Spiewak
@djspiewak
Apr 27 2016 04:49
I know the second is possible, but I also know it's pretty hard. the first sounds doable, but dodgy
oh god in heaven…
actually
Timothy Perrett
@timperrett
Apr 27 2016 04:49
actually, come to think of, cant we leverage that
Daniel Spiewak
@djspiewak
Apr 27 2016 04:49
maybe that will help
Timothy Perrett
@timperrett
Apr 27 2016 04:49
yeah
Daniel Spiewak
@djspiewak
Apr 27 2016 04:49
yeah seriously
just toss it in a sys prop
SHIP IT™
Timothy Perrett
@timperrett
Apr 27 2016 04:49
exactly what i was thinking :P
Daniel Spiewak
@djspiewak
Apr 27 2016 04:50
:-)
same terrible idea at the same instant
the question is how do you do local building?
Timothy Perrett
@timperrett
Apr 27 2016 04:50
@djspiewak its like every day at the office slack channel :P
Daniel Spiewak
@djspiewak
Apr 27 2016 04:50
like, you need to be able to at least run the unit tests
lol. true story
Timothy Perrett
@timperrett
Apr 27 2016 04:50
holy cow, this would actually work - travis allows you to propage env vars as part of the matrix
Daniel Spiewak
@djspiewak
Apr 27 2016 04:50
I guess we can just have a setting that users flip in the sbt console
yeah, exactly
Timothy Perrett
@timperrett
Apr 27 2016 04:50
yeah exactly
Daniel Spiewak
@djspiewak
Apr 27 2016 04:51
it's mad
mad as in "awesome"
Timothy Perrett
@timperrett
Apr 27 2016 04:51
its like everyday
Daniel Spiewak
@djspiewak
Apr 27 2016 04:51
lol
so yeah, then separate source dirs which are gated by whether or not 7.2 vs 7.1 is on the classpath
Timothy Perrett
@timperrett
Apr 27 2016 04:52
yeah - super easy
Daniel Spiewak
@djspiewak
Apr 27 2016 04:52
(well, gated by the upstream property which governs that)
yeah
(by property I mean setting key)
Timothy Perrett
@timperrett
Apr 27 2016 04:52
dont worry, i follow ;-)
Daniel Spiewak
@djspiewak
Apr 27 2016 04:52
:-)
mad like a fox
Gastón Tonietti
@ktonga
Apr 27 2016 04:55
separate src folders in your case makes sense as you have almost no API contact with scalaz directly in knobs and it is src compatible in the case of ermine
Timothy Perrett
@timperrett
Apr 27 2016 04:55
right, for ermine its a slam dunk as we’ll just roll a new version
Gastón Tonietti
@ktonga
Apr 27 2016 04:56
so you're gonna upgrade it? need any help from me?
Timothy Perrett
@timperrett
Apr 27 2016 04:58
@ktonga im going to roll ermine right now, and then we can figure out what to do about knobs - i guess we’ll have to essentially duplicate sources for some stuff
not the end of the world though
Daniel Spiewak
@djspiewak
Apr 27 2016 05:00
@timperrett you should only have to duplicate sources where you actually have source-incompatible scalaz usage
Timothy Perrett
@timperrett
Apr 27 2016 05:00
yeah
Daniel Spiewak
@djspiewak
Apr 27 2016 05:00
since you're forcing a rebuild, any usages that are source compatible will just work
so I guess… anything where you call runAsync :-)
there really aren't many source-incompatible changes between 7.1 and 7.2
Gastón Tonietti
@ktonga
Apr 27 2016 05:00
mostly in testing where you are using some runs
Timothy Perrett
@timperrett
Apr 27 2016 05:01
yeah
Daniel Spiewak
@djspiewak
Apr 27 2016 05:01
you're probably best off defining a def runT[A](t: Task[A]): A = t.run function in the scalaz-specific source dir
Gastón Tonietti
@ktonga
Apr 27 2016 05:01
you can also move al that sh*t so some sort of Utility (sorry) object
Daniel Spiewak
@djspiewak
Apr 27 2016 05:01
and just make it unsafeRun for 7.2
@ktonga exactly :-)
progress!
Gastón Tonietti
@ktonga
Apr 27 2016 05:17
@timperrett since all the changes are in tests and one in zookeeper module (which I'm not using) I think that just forcing ermine 0.3 in my build will do it
Timothy Perrett
@timperrett
Apr 27 2016 05:18
@ktonga im just working on the main knobs upgrade now - i have the travis matrix done, just working on the compatibility and the build
Gastón Tonietti
@ktonga
Apr 27 2016 05:19
oh wow so I can wait :)
Timothy Perrett
@timperrett
Apr 27 2016 05:37
@djspiewak think adding the source directory is enough?
its not picking it up for some reason - i mean, it sees the folder as a new source dir, but it doesnt compile sources there
Rob Norris
@tpolecat
Apr 27 2016 05:37
oh lord no I don't use maven, what the hell is wrong with you
Timothy Perrett
@timperrett
Apr 27 2016 05:37
@tpolecat LOL i must have miss-heard
@djspiewak dont worry i got it :)
Rob Norris
@tpolecat
Apr 27 2016 05:42
i use osgi, so maybe that's what you were thinking of
the maven of service achitectures
Timothy Perrett
@timperrett
Apr 27 2016 05:42
LOL
hilarious @tpolecat
Rob Norris
@tpolecat
Apr 27 2016 05:44
and maven is the jira of build tools
i should make a chart
Timothy Perrett
@timperrett
Apr 27 2016 05:46
@tpolecat i’d like to see that chart
Timothy Perrett
@timperrett
Apr 27 2016 05:59
@ktonga @djspiewak oncue/knobs#29
and sure enough, it explodes the matrix: https://travis-ci.org/oncue/knobs/builds/126022971
WIN
Gastón Tonietti
@ktonga
Apr 27 2016 06:03
Great. SBT should support it. I'll implement the sbt-matrix-plugin someday.
Timothy Perrett
@timperrett
Apr 27 2016 06:05
once it builds, we can get it merged
Colt Frederickson
@coltfred
Apr 27 2016 16:05
Is there a specific reason that knobs depends on a super old version of scalaz stream? If I updated to 0.8.a in a PR would that be accepted? I expect we'd want to bump to either 4.x or 3.4 depending on which versioning scheme knobs subscribes to.
Colt Frederickson
@coltfred
Apr 27 2016 16:18
Oh I see there's been a bit of work done and that the version is 3.6 now... I'm a bit confused. If anyone could shed some light on the plan, I'd be interested :D
Timothy Perrett
@timperrett
Apr 27 2016 16:44
@coltfred im actually trying to cut a release this morning that supports both versions at the same time
@coltfred having some issues with automating the build - the artifacts keep getting named incorrectly, and i need to go attend to some other things. It’s like 99% of the way thre, so if you want to figure out what the last issue is that would be good
ideally i want automated release from travis - its pretty much there, but its just a stupid naming thing
@coltfred dont depen on 3.6 right now :)
Timothy Perrett
@timperrett
Apr 27 2016 16:49
@coltfred yeah, its not honouring the artifactName i’ve configured: https://travis-ci.org/oncue/knobs/jobs/126156912#L1654
Colt Frederickson
@coltfred
Apr 27 2016 16:51
Is there really no way using knobs.TypesafeConfig to load a custom HOCON file?
Timothy Perrett
@timperrett
Apr 27 2016 16:52
what?
Colt Frederickson
@coltfred
Apr 27 2016 16:52
Timothy Perrett
@timperrett
Apr 27 2016 16:53
right, but isnt that dictate by that silly system property they use? -Dapp.config or something?
Colt Frederickson
@coltfred
Apr 27 2016 16:53
oh...
I need to load lots of config files.
Timothy Perrett
@timperrett
Apr 27 2016 16:53
yeah, we’re only using that module for a few use cases as points of integration, so we dont typically have that issue. Happy to have a PR for it though
there are some parts of the typesafe config stuff thats just incompatible with what knobs is doing
but it should be possible to make that more flexible
we’re slowly moving away from typesafe config all over the place - its been surprisingly hard to remove; all new stuff we’ve been doing with knobs since we built it, but some TC stuff still hangs around
Colt Frederickson
@coltfred
Apr 27 2016 16:56
Oh ok.
Timothy Perrett
@timperrett
Apr 27 2016 16:57
@coltfred does that help
Colt Frederickson
@coltfred
Apr 27 2016 16:59
Well, it sheds light on the issue. I can certainly write my own ResourceBox => Config for TC. We also have [{"key": "value"}] which was asked about here: oncue/knobs#23
Timothy Perrett
@timperrett
Apr 27 2016 17:00
yeah, we dont want Map[String,String], but if someone wants to send a PR with some support for that, we’d certainly give it good consideration. Typically we’ve found that you dont want data as config keys, as it leads to abuse
Murph Murphy
@skeet70
Apr 27 2016 17:01
ex:
lookups: [
  {
    lookup-type: ${query-type.sql},
    datasource: ${datasource.ui},
    format: ${format.simple-json},
    columns: ["campaign_id", "campaign_type"],
    query: "SELECT campaign_id, campaign_type FROM campaigns;",
    filename: "campaign_id_to_type"
  },
  ...
]
Colt Frederickson
@coltfred
Apr 27 2016 17:01
So it's really List of case class.
Timothy Perrett
@timperrett
Apr 27 2016 17:02
the pasted example above is not exhibiting this problem, as it doesnt use data as keys.
Colt Frederickson
@coltfred
Apr 27 2016 17:02
Seems like HList decoders might work well for that.
Timothy Perrett
@timperrett
Apr 27 2016 17:02
possibly
we try to keep things simple - many config systems just become too powerful
Colt Frederickson
@coltfred
Apr 27 2016 17:03
What's the way to decode the above using knobs? it's a list of subconfigs.
Murph Murphy
@skeet70
Apr 27 2016 17:03
So in your current system how would you do the above?
Timothy Perrett
@timperrett
Apr 27 2016 17:06
well, i probally wouldnt set things up like that tbh, but if i did, then it might be possible to wrangle something with Configured
Murph Murphy
@skeet70
Apr 27 2016 17:07
So if you had a usecase like that, what would be your preferred setup? I'm definitely amenable to doing in a better/different way. I'm migrating from a Typesafe config where that was supported and seemed to really naturally fit the usecase.
Timothy Perrett
@timperrett
Apr 27 2016 17:10
so first and foremost, i’d like to say that people should use the tool that best fits their use case - if thats knobs, great, if its TC then thats fine too; we’re not trying to make an uber solution that solves all use cases. For us, that kind of “config” is really application data, and we’d not “configure” it like that more than likley. Config for us is usually a discrete set of data that informs the application about edge of world setup and provides default arguments and so forth - it is specifically not too powerful so that people dont do crazy config that is more like a program with it, as we had previously accumulated quite a bit of debt with people doing that
like, when i look at that, and know that you’ll load that at runtime, but its checked into source, i’d just prefer to do it in scala and roll a new release of the code, as im sure you probally do with the config
at which point its moot that its in config file or code, and code is typechecked
as one example
other option is to load that from a file, or datastore, or whatever
i have to duck out for some time to do other work - hope this helps though, and if knobs is a fit for you, awesome :-)
Murph Murphy
@skeet70
Apr 27 2016 17:13

The config there would be outside the jar, and able to be modified by ops if necessary without redeployment. Which was the idea w/rt making it a config instead of in code.

I agree though. That's all I really needed to know, knobs probably isn't going to fit our usecase. Thanks for your help!

Timothy Perrett
@timperrett
Apr 27 2016 17:14
@skeet70 yeah, we specifically never do that - ops do not modify configuration files in our org, we optiomized the deploy pipeline so config == code, and getting a corrected version is automated and fast :-) we kept burning ourselves with runtime modification of anything. Immutable infrastructure FTW
Murph Murphy
@skeet70
Apr 27 2016 17:15
Definitely agree. I'm over in a less good place though infrastructure wise :D. We're just introducing FP to the organization, so it needs to place nice with other processes.
Timothy Perrett
@timperrett
Apr 27 2016 17:15
yeah totally makes sense
dont get me wrong - this stuff is hard
getting a slick infrastructure is time consuming and requires dedicated resources
so its not cheap, not fast, and its still hard