These are chat archives for CommBank/maestro

17th
Apr 2015
Conrad Parker
@kfish
Apr 17 2015 01:53
i've updated CommBank/ebenezer#100 to take into account stephan's comments
Stephan Hoermann
@stephanh
Apr 17 2015 02:26
@kfish, @toddmowen can you please make sure you have a look at the PRs being raised just to familarise yourselves with the code base, etc.
Sam Roberts
@SamRoberts
Apr 17 2015 02:52

@stephanh running load executions which do no IO (use only IterablePipe and run with toIterableExecution) with the following empty configuration:

val config = Config.empty
val mode   = Test(_ => None) // the Mode

Can you think of any issues with this?

Sam Roberts
@SamRoberts
Apr 17 2015 03:04
ha, despite my efforts to minimize setup time by preparing thread pools and so forth outside the benchmark test, the "load execution pipeline" benchmark is completely dominated by a constant factor of approx 200 ms. I'm going to try reducing the default poling intervals, as we do in ThermometerSpec, but if that doesn't get rid of the constant factor I don't think I can get a useful "entire pipeline" benchmark today
no, reducing the polling interval doesn't seem to help
Luke Williams
@shmookey
Apr 17 2015 06:00
hey guys, i need to go and pick up my partner from the airport so i don't have time to get this code as pretty as i'd like it, but, well, here you are: a tentative transformer PR: CommBank/maestro#369
Sam Roberts
@SamRoberts
Apr 17 2015 06:01
I'm still waiting for any reviewers for #368 :smile:
Luke Williams
@shmookey
Apr 17 2015 06:01
oh sorry! i would take a look at it but i really have to run
Sam Roberts
@SamRoberts
Apr 17 2015 06:01
(I know Friday's not necessarily a good day)
Vineeth Varghese
@vineethvarghese
Apr 17 2015 06:01
@SamRoberts I will take a look now
Sam Roberts
@SamRoberts
Apr 17 2015 06:01
no worries @shmookey , thanks @vineethvarghese
Todd Owen
@toddmowen
Apr 17 2015 06:40
#370 (Adds HdfsUtils and deprecates Guard) is ready for review, but can wait until next week, given the sudden deluge of Friday afternoon PRs (…two).
Rowan Davies
@rowandavies
Apr 17 2015 06:58
@stephanh - when you get a chance, can I get your opinion on code that uses a “by-year” directory and has the partitioning specialized to String.