These are chat archives for CommBank/maestro

13th
Apr 2015
Jithin John
@jithinjohn
Apr 13 2015 00:48
Hi I have modified MaestroConfig to support command line args for hive database prefix. Could some one please review and confirm whether it is okay . #360
Sam Roberts
@SamRoberts
Apr 13 2015 01:00
thanks @jithinjohn , it's really great that you have stepped up to do that
Luke Williams
@shmookey
Apr 13 2015 01:08
@stephanh @SamRoberts just to keep you up to date, this is where the transform syntax is at for now:
@transformer def foobarTransform (foo: Foo, bar: Bar): FooBar = (
  thatInt    := foo.thisInt,
  thatString := bar.thisString
)
Sam Roberts
@SamRoberts
Apr 13 2015 04:36
Can someone please review #361 ?
Sam Roberts
@SamRoberts
Apr 13 2015 04:51
@shmookey would it be possible to do @transformer val myFunc = (foo: Foo, bar: Bar): FooBar = ...?
what sort of error message does the user get if they apply @transformer somewhere invalid?
Sam Roberts
@SamRoberts
Apr 13 2015 04:57
it almost feels like we can avoid custom syntax en]
entirely by generating a method with named arguments and default values
although I have no fully formed ideas
and I'm not sure what @shmookey would think of redoing the syntax again :smile:
Stephan Hoermann
@stephanh
Apr 13 2015 04:59
It's still custom syntax because the function body is a list of transformations
@SamRoberts I don't think what you are proposing is legal Scala syntax and won't compile. What you could do is @transformer val myFunc: (Foo, Bar) => FooBar = (foo: Foo, bar: Bar) => ...
I like the def syntax better.
Sam Roberts
@SamRoberts
Apr 13 2015 05:01
Hrmm, that's true. I think if you avoid custom syntax this is the best you can do: (foo: Foo, bar: Bar) => generated(thatInt = foo.thisInt, thatString = bar.thisString)
Oh, my question about val is not related to my comments about custom syntax!
Sorry for the confusion
That was just asking how flexible the @transfomer annotation is
And then, as an entirely separate comment, wondering whether you actually need custom syntax at all
Stephan Hoermann
@stephanh
Apr 13 2015 05:05
The original transformer macro has no custom syntax but a bit of boilerplate. However, with custom syntax you can save on the boilerplate.
Sam Roberts
@SamRoberts
Apr 13 2015 05:08
yes, I mean, looking at @shmookey's latest syntax, I was wondering whether you can have your cake and eat it to. Which turned into the (a: A, b: B) => generated_body(modifications) suggestion above, where the macro is expanded inside the function body.
Stephan Hoermann
@stephanh
Apr 13 2015 05:13
Generated body itself needs to produce a function (A, B) => AB in order to do the default transformations.
When you are in generated_body you probably don't want to look at the outer scope to determine that you are inside a function and what the function parameters are.
Sam Roberts
@SamRoberts
Apr 13 2015 05:18
Yeah, I've been thinking about that. Can't see any way around it either. I mean, the generated body doesn't have to call a generated function with named arguments and defaults, but then you are back to using custom syntax to declare overrides. I don't think looking at the outer scope has to be a big deal: you can always insist that the immediate context is the function declaration, but it's not exactly nice either.
So yeah, with the generated_function inside generated_body, it might be a case of the cure being worse than the disease
Luke Williams
@shmookey
Apr 13 2015 05:29
personally i don't find custom syntax to be such a harmful disease if used judiciously
Luke Williams
@shmookey
Apr 13 2015 05:34
querying the containing scope would violate my sense of decency though :P
Sam Roberts
@SamRoberts
Apr 13 2015 05:34
haha
Sam Roberts
@SamRoberts
Apr 13 2015 05:46
can someone please review #362 ? The commit itself is trivial, it's more the possibility for backwards incompatibility which concerns me. Can anyone think of any possible issues (e.g. upgrading maestro and running a job "after" the previous job)?
Rowan Davies
@rowandavies
Apr 13 2015 05:59
@SamRoberts Looking at #362.
Todd Owen
@toddmowen
Apr 13 2015 07:11
Can someone please review #363? It's my first PR, no change to existing code, only new tests.
Rowan Davies
@rowandavies
Apr 13 2015 08:09
@toddmowen One of your tests (“expandPaths: skips processed dirs”) is failing in the Travis CI build. Follow the “Details” link from the github page for #363.
Stephan Hoermann
@stephanh
Apr 13 2015 22:59
Sorry guys I'm going to be about 5 minutes late for the stand up
Luke Williams
@shmookey
Apr 13 2015 23:22
I'm going to work from home this morning, maybe all day, I just really need to be able to smash through a growing pile of stuff e.g. hack day organisation with minimal distraction
please dial me in for the standup though!
Rowan Davies
@rowandavies
Apr 13 2015 23:30
5 mins late
Stephan Hoermann
@stephanh
Apr 13 2015 23:50
I forgot to mention at the stand up that I will be working from home tomorrow.