These are chat archives for CommBank/maestro

7th
Apr 2015
Stephan Hoermann
@stephanh
Apr 07 2015 00:06
@vineethvarghese how is your build problem going?
Vineeth Varghese
@vineethvarghese
Apr 07 2015 00:10
Gavin Whyte
@gavinwhyte
Apr 07 2015 00:11
Travis-ci has had capacity issues last week, which i think is resolved.
Unfortunately we are still experiencing team city build issues, i am trying to escalate it this morning.
Sam Roberts
@SamRoberts
Apr 07 2015 00:26
thanks for all your work @gavinwhyte
Gavin Whyte
@gavinwhyte
Apr 07 2015 00:47
:)
Rowan Davies
@rowandavies
Apr 07 2015 00:55
@SamRoberts @Stephanh My impression is that Kyro automatically serializes in more situations, but is less predicatable and generally requires some work to deal with specific kinds of tricky situations where it fails. Some of the serialization related scalding issues seem worth looking at - e.g., twitter/scalding#1154 includes:
At twitter, the pattern is to set up a base job (we call TwitterJob), which overrides 
the config to add serializations to the kryo serializer:

https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/Config.scala#L124

We add protobuf, scrooge-scala, and tbase thrift (all in chill).
Gavin Whyte
@gavinwhyte
Apr 07 2015 01:14
Unfortunately Stacc team is delayed with regards to fixing this issue. Saga continues.

1) In order to get the most value from Driven we need the ability to set flow names and add application tags from maestro. In the context of the example project in maestro, where can we do this?

Here are examples of setting flow names and application tags in Scalding:
Flow name:
```implicit val mode = Hdfs(new JobConf())
implicit val flowDef = new FlowDef
flowDef.setName("Flow123")
val result = myFunctionThatTakesFlowDefAndMode(flowDef, mode)
mode.newFlowConnector(config).connect(flowDef).complete

unfortunately the Driven team needs help
Application Tags:
class MyJob(args: Args) extends Job(args) {
override def config: Map[AnyRef, AnyRef] = {
val props = new Properties()
AppProps.addApplicationTag(props, "mytag")
super.config ++ props.asJava
}
}
Stephan Hoermann
@stephanh
Apr 07 2015 01:23
@gavinwhyte I thought that the concurrent guys would do a rough draft of integrating it with Maestro
Gavin Whyte
@gavinwhyte
Apr 07 2015 01:23
unfortunately they are stuck
Sam Roberts
@SamRoberts
Apr 07 2015 01:24
is this because they haven't integrated with the Execution api yet?
Gavin Whyte
@gavinwhyte
Apr 07 2015 01:24
@stephanh unfortunately they are stuck, with the draft i am trying to get them across the line.
@SamRoberts No they havent
Sam Roberts
@SamRoberts
Apr 07 2015 01:25
As part of the one execution we will have many flows which really need to be grouped to make a lot of sense
Gavin Whyte
@gavinwhyte
Apr 07 2015 01:26
Thats where the tagging comes in
I am trying to get the first draft accross the line and then hand it to you guys for the actual loads.
Stephan Hoermann
@stephanh
Apr 07 2015 01:28
@gavinwhyte do they have any understanding / have worked with the execution API before?
Sam Roberts
@SamRoberts
Apr 07 2015 01:29
maestro is directly exposing Executions, so the first thing to do is figure out how to accomplish this with the Execution API, and never mind Maestro at all
Gavin Whyte
@gavinwhyte
Apr 07 2015 01:32
I don't quiet think they get that, I will have a chat with them tonight, ensure they understand this.
Stephan Hoermann
@stephanh
Apr 07 2015 01:34
Ok
Sam Roberts
@SamRoberts
Apr 07 2015 01:51
Oh. My. Gosh.
Well, Java serialization sucks
that class not found exception? The Java ObjectInputStream gets the classloader with which to load classes by travelling up the stack and taking the classloader from the first class whose getClassLoader method does not return null
so most core Java types, for instance, will be loaded with the bootstrap classloader and getClassLoader will return null
anyway, in the case of Field, the first class whose getClassLoader does not return null is scala.collection.immutable.$colon$colon
(the list cons constructor)
it turns out that this classloader for this core scala class is the father of the classloader of our classes
so it can't find our classes
and afaict there's not much we can do about this
Sam Roberts
@SamRoberts
Apr 07 2015 01:57
(although chill has an "Externalizer" somewhere above this in the call stack, so it may be we could change some hook that chill adds to fix this, but at that point we should probably just be making kyro work)
Sam Roberts
@SamRoberts
Apr 07 2015 02:07
(to elaborate slightly on the last point, this is a problem when reading from an object stream, not writing to it as you would expect to be the case when serializing our objects. Why are we reading from an object stream? I don't know the details, but it is chill's externalizer which starts doing it. None of this changes the fact that the Java serialization framework is using an incredibly hacky method for getting the "users" classpath, and I doubt we can rely on it 100%.)
Jonathan Merritt
@lancelet
Apr 07 2015 02:20
@SamRoberts I didn't follow all of the above, but would it help to implement your own serializer? @mjhopkins created one for org.joda.time.DateTime that is mixed-in to Jobs:
https://github.com/CommBank/eventually/blob/master/eventually-core/src/main/scala/au/com/cba/omnia/eventually/core/DateTimeSerialization.scala
Sam Roberts
@SamRoberts
Apr 07 2015 02:24
yeah, but that's a kyro serializer, so that's a different kettle of fish entirely. Kyro is failing on maestro's validators, and I haven't even begun to look at that yet. I was focusing on Java serialization because that's what our attempts at solving serialization issues have actually been addressing (although I don't think we knew that, as both options are tried under the hood, and we were just changing code until it worked)
Rowan Davies
@rowandavies
Apr 07 2015 02:50
@SamRoberts I think there's a good chance the Kryo failure on validators can be resolved without too much pain. Most likely it is of a similar nature to the Kryo tuple issues or thrift issues mentioned in the scalding issues. If Twitter went that way, at least we know it can be done.
Sam Roberts
@SamRoberts
Apr 07 2015 03:22
yeah, after I take a look at the maestro issue mentioned earlier today, I'll look at getting Kryo up and running
Rowan Davies
@rowandavies
Apr 07 2015 03:24
@SamRoberts It's weird that the issue occurs during while reading the object stream. Presumably this is while reconstructing objects on the hadoop nodes that are local clones of objects previously created where the job started. It sounds like something basic is broken with the standard serialization, perhaps when support for Kryo was added to scalding. If we want to consider the Kryo route, I guess we can look what twitter do in TwitterJob.
Sam Roberts
@SamRoberts
Apr 07 2015 03:26
no, it's actually while serializing the objects on the client. I obviously did a poor job of explaining that
here, perhaps a stack trace snipped will tell a better story:
Rowan Davies
@rowandavies
Apr 07 2015 03:27
Oh - I think I get it.
It's while converting a Stream[Object] into a Stream[byte]
Sam Roberts
@SamRoberts
Apr 07 2015 03:28
java.lang.ClassNotFoundException: au.com.cba.omnia.maestro.core.data.Field
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:274)
    at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:625)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)

        <snip>

    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at com.twitter.chill.Externalizer.probeJavaWorks(Externalizer.scala:117)
    at com.twitter.chill.Externalizer.javaWorks(Externalizer.scala:102)
    at com.twitter.chill.Externalizer.writeJava(Externalizer.scala:165)
    at com.twitter.chill.Externalizer.maybeWriteJavaKryo(Externalizer.scala:182)
    at com.twitter.chill.Externalizer.writeExternal(Externalizer.scala:189)
    at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)

        <snip>

    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at cascading.flow.hadoop.util.JavaObjectSerializer.serialize(JavaObjectSerializer.java:57)
    at cascading.flow.hadoop.util.HadoopUtil.serializeBase64(HadoopUtil.java:282)

In the above stack trace, you can see hadoop using the java object api to serialize an object, you can see the chill Externalizer being inserted into the logic, you can see the Externalizers start to read from an ObjectInputStream, and you can see the ObjectInputStream try and fail to load a class because it picked up the wrong classpath loader form the stack trace

But I accidentally omitted the line containing the class where it picked up the wrong classloader from

Rowan Davies
@rowandavies
Apr 07 2015 03:38
Right - I think I agree with that interpretation of the stack trace. But, it's bizarre to choose the class loader that way - at one point a few years ago I think I understood why it's done that way.
Vineeth Varghese
@vineethvarghese
Apr 07 2015 03:53
Has anyone seen the following error when travis run ci/gh-pages.sh :
[error] /home/travis/build/CommBank/etl-controller/target/scala-2.10/src_managed/main/info.scala:2: VersionInfo is already defined as object VersionInfo
[error] object VersionInfo {
[error]        ^
[error] /home/travis/build/CommBank/etl-controller/test/target/scala-2.10/src_managed/main/info.scala:2: VersionInfo is already defined as object VersionInfo
[error] object VersionInfo {
[error]        ^
[info] No documentation generated with unsucessful compiler run
[error] two errors found
[error] (all/scalaunidoc:doc) Scaladoc generation failed
[error] Total time: 6 s, completed Apr 7, 2015 3:44:10 AM
Rowan Davies
@rowandavies
Apr 07 2015 03:57
@SamRoberts Also bizarre is that the class Field isn't already loaded - since presumably we reach this because we have an instance of Field to serialize.
Vineeth Varghese
@vineethvarghese
Apr 07 2015 03:58
@rowandavies Not sure of the details but the exception stachtrace just means that classloader used for deserialization can't see that class.
Another classloader might have loaded it but not the classloader used here
Sam Roberts
@SamRoberts
Apr 07 2015 03:59
That's right. In this case we end up using the parent of the classloader used to load all our classes.
Stephan Hoermann
@stephanh
Apr 07 2015 04:00
@vineethvarghese I haven't seen it before but I can hypothesise. The uniform unique version plugin creates a scala object with the version number. I think this object is created for each subproject and put in the package name which is the second arugment to uniform.project. In the case of etl-controller both subprojects have the same package name.
As an important side note though we can't create github pages for private repos since github pages are always public.
Vineeth Varghese
@vineethvarghese
Apr 07 2015 04:02
@stephanh hmm...good point. What I really want is the scala docs
Stephan Hoermann
@stephanh
Apr 07 2015 04:03
Yes, we use github pages to host them but we can't do that with private repos.
Vineeth Varghese
@vineethvarghese
Apr 07 2015 04:39
@SamRoberts Did you update etl.util to point to the maestro version with the header and trailer parsing?
Sam Roberts
@SamRoberts
Apr 07 2015 04:40
no, can't touch util.etl until the team city issues are resolved
Vineeth Varghese
@vineethvarghese
Apr 07 2015 04:40
That's good new
Sam Roberts
@SamRoberts
Apr 07 2015 04:40
oh, really? since when?
Vineeth Varghese
@vineethvarghese
Apr 07 2015 04:40
:)
Sam Roberts
@SamRoberts
Apr 07 2015 04:41
wasn't @sujeshce complaining about team city issues just 15 mins ago?
Vineeth Varghese
@vineethvarghese
Apr 07 2015 04:41
it is not :).
Sam Roberts
@SamRoberts
Apr 07 2015 04:41
ok, so team city is still broken
and there's nothing I can do with util.etl?
Gavin Whyte
@gavinwhyte
Apr 07 2015 05:30
Team City is still broken, i have just escalated the process.
i shall keep you updated once its fixed
Stephan Hoermann
@stephanh
Apr 07 2015 05:32
Are we actually sure it is still broken?
Gavin Whyte
@gavinwhyte
Apr 07 2015 05:33
yep just check
I should have said Artifactory has missing dependencies
Stephan Hoermann
@stephanh
Apr 07 2015 05:38
@gavinwhyte which build and what dependencies are missing?
Gavin Whyte
@gavinwhyte
Apr 07 2015 05:39
[09:51:15][Step 2/5] [warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[09:51:15][Step 2/5] [warn]     ::          UNRESOLVED DEPENDENCIES         ::
[09:51:15][Step 2/5] [warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[09:51:15][Step 2/5] [warn]     :: org.apache.avro#avro-mapred;1.7.6-cdh5.2.4: not found
[09:51:15][Step 2/5] [warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[09:51:15][Step 2/5] [warn] 
[09:51:15][Step 2/5] [warn]     Note: Unresolved dependencies path:
[09:51:15][Step 2/5] [warn]         org.apache.avro:avro-mapred:1.7.6-cdh5.2.4
[09:51:15][Step 2/5] [warn]           +- org.apache.sqoop:sqoop:1.4.5-cdh5.2.4
[09:51:15][Step 2/5] [warn]           +- au.com.cba.omnia:parlour_2.10:1.8.1-20150326035955-18bc8d9
[09:51:15][Step 2/5] [warn]           +- au.com.cba.omnia:maestro-core_2.10:2.5.0-20150402015625-7c6bfa5 (/F/TCBuildAgent/buildAgent/work/7f389536440e104a/project/build.scala#L36)
[09:51:15][Step 2/5] [warn]           +- au.com.cba.omnia:maestro-macros_2.10:2.5.0-20150402015625-7c6bfa5 (/F/TCBuildAgent/buildAgent/work/7f389536440e104a/project/build.scala#L36)
[09:51:15][Step 2/5] [warn]           +- au.com.cba.omnia:maestro_2.10:2.5.0-20150402015625-7c6bfa5 (/F/TCBuildAgent/buildAgent/work/7f389536440e104a/project/build.scala#L36)
[09:51:15][Step 2/5] [warn]           +- au.com.cba.omnia:etl-util_2.10:PR40-1.4.0-20150407094940-f32c215
[09:51:20][Step 2/5] sbt.ResolveException: unresolved dependency: org.apache.avro#avro-mapred;1.7.6-cdh5.2.4: not found
from @rakeshrnair
Rakhesh Rajendran Nair
@rakheshrnair
Apr 07 2015 05:39
@stephanh the util-etl PR is having the issue
the team city build is failing with the above issue
Stephan Hoermann
@stephanh
Apr 07 2015 05:40
Yeah that is not a problem with artifactory. It is how it was designed to work for omnia http://knowit.cba/display/OMNIA/Artifactory+Repositories+Overview
Sam Roberts
@SamRoberts
Apr 07 2015 06:27
@stephanh I have added a serialization related comment in #142 . I think the original issue might be outdated now, but there is definitely lots to note about serialization with the new Execution API. I don't have any clear fix yet. We will probably have to start looking seriously at kryo and making sure it can serialize all our classes.
Gavin Whyte
@gavinwhyte
Apr 07 2015 06:27
Ok i assume that all builds are working
i will close ticket
Vineeth Varghese
@vineethvarghese
Apr 07 2015 07:13
@gavinwhyte Isn't artifactory a proxy repo, i.e. if a build request for a dependency, artifactory will download it from the internet for the build. It didn't quite work for a build, hence the question.
Gavin Whyte
@gavinwhyte
Apr 07 2015 09:30
@vineethvarghese you are correct
let me know which dependency didnt download
Sam Roberts
@SamRoberts
Apr 07 2015 11:09
I'll be flying down tomorrow morning, so won't be available for the standup. My status:
Sam Roberts
@SamRoberts
Apr 07 2015 11:17
  • Team city is building our repos again: merged the hive-udfs changes to master. Will add any further changes requires to util.etl tomorrow. Then look at etl-plugin.
  • Merged the Splitter.csv functionality to master
  • Investigated serialization issues:
    • Learned a bunch of things about how scala compiles code, which you need to know to understand serialization issues.
    • Learned how to get additional information about serialization failures.
    • Learned a bit more about how Java serialization works.
    • Still need to learn more about kryo serialization.
    • Found some existing serialization failures in maestro, and created an issue for them.
    • Would like to record some sort of "how-to" for debugging serialization issues, but not really sure where to put it.
Sam Roberts
@SamRoberts
Apr 07 2015 11:29
can someone please review #354 ?
Vineeth Varghese
@vineethvarghese
Apr 07 2015 12:25
@gavinwhyte That would be com.opencsv:opencsv:3.3
Sam Roberts
@SamRoberts
Apr 07 2015 12:26
that's the new dependency I just added
Vineeth Varghese
@vineethvarghese
Apr 07 2015 12:26
true...but artifactory should just pull that down without bothering me
Sam Roberts
@SamRoberts
Apr 07 2015 12:27
what repos does it mirror?
Vineeth Varghese
@vineethvarghese
Apr 07 2015 12:27
I am not sure...which is why I wanted to Gavin although opencvs should be there in the popular repos out there
Sam Roberts
@SamRoberts
Apr 07 2015 12:28
yeah
Stephan Hoermann
@stephanh
Apr 07 2015 23:01
I'm going to be about 5 minutes late. Sorry