These are chat archives for CommBank/maestro

25th
May 2015
Vineeth Varghese
@vineethvarghese
May 25 2015 00:42
Can someone please review CommBank/etl-controller#25
Shobin Joseph
@josephshobin
May 25 2015 02:13
Can someone pls review CommBank/etl-controller#26
Stephan Hoermann
@stephanh
May 25 2015 02:36
I have to leave early today got a dentists appointment,
Conrad Parker
@kfish
May 25 2015 03:53
any suggestions on a scala/java regex library to use?
i just want something i can use to filter a List[Path]
(preferably in a way that's compatible with the Hadoop library's globStatus function :)
Vineeth Varghese
@vineethvarghese
May 25 2015 04:46
Regex support is available in java which is most likely what hadoop is using
Todd Owen
@toddmowen
May 25 2015 04:47
actually I looked at the code recently and I think that hadoop itself implements the pattern matching behind globStatus
not sure whether that’s something @kfish could call directly or not
Conrad Parker
@kfish
May 25 2015 04:50
what i really want is 1) a list of files matching globPattern, and 2) a list of all child dirs
so currently i'm calling Hdfs.files() twice, once with globPattern, once with globPattern="*"
then doing filterM(isDirectory) on the latter
Conrad Parker
@kfish
May 25 2015 04:52
so i could either replace the first call with a regexFilter(globPattern) on the full listing, or use a listDirs against the namenode to get the child dirs
Conrad Parker
@kfish
May 25 2015 04:57
@dkristian thanks
Rowan Davies
@rowandavies
May 25 2015 08:16
@shmookey Any progress on the “all fields null” issue for #377 ? I was just reviewing etl.commfx-foreignexchange which currently works around this issue, let me know if more details would be useful.
(Alas, stash is down currently.)
Conrad Parker
@kfish
May 25 2015 20:35

I'll miss this morning's standup.
Yesterday:

  • merged the ebenezer cdh5.4.2-compat tests
  • updated thrift-arbitrary, the travis test is still failing at scaladoc on quasiquotes (which I thought would work in 2.11?), need to investigate
  • refactored permafrost files/find (merged the find functionality into the files method directly, as a "recursive" flag)

Today I'll keep working on these outstanding PRs (plus the viewHive overwrite, that uses the permafrost change).

Vineeth Varghese
@vineethvarghese
May 25 2015 22:44
I have to work from home today. Please call me for the standup
Stephan Hoermann
@stephanh
May 25 2015 23:49
I have changed codacy so that it doesn't raise PR comments anymore but still gives a status at the bottom like travis. Please let me know how that goes.
Tin Pavlinic
@triggerNZ
May 25 2015 23:50
cool thanks
in general, how much should code comply with codacy?
Stephan Hoermann
@stephanh
May 25 2015 23:54
It's at your discretion treat it as a dumb assistant that might flag potential issues to you and you decide what to do with the information.
Tin Pavlinic
@triggerNZ
May 25 2015 23:54
okay cool
What are your thoughts about adding something like scalastyle into the build to enforce some of the style issues that came up in that PR?
Stephan Hoermann
@stephanh
May 25 2015 23:59
Codacy should be doing some of that. I don't really have any strong thoughts on it. Most of the contributors to Maestro are regular and after the first or second PR style isn't really too much of an issue anymore.