These are chat archives for CommBank/maestro

28th
Apr 2015
Luke Williams
@shmookey
Apr 28 2015 01:10
def getCustomConnMan: Option[String] this method promised to help me form a connection but ended up stealing all my money and left me with None
ugh sorryt lol
sounded better in my head
Conrad Parker
@kfish
Apr 28 2015 02:23
is issues.scala-lang.org down?
Jonathan Merritt
@lancelet
Apr 28 2015 02:26
@kfish nah... we just can't connect to it from within the bank network. Makes complete sense, right? :-P
Conrad Parker
@kfish
Apr 28 2015 02:26
srsly?
Jonathan Merritt
@lancelet
Apr 28 2015 02:28
Yup (try accessing it on a phone - that's what @mjhopkins is always doing). I think Mark has mentioned it as a problem a few times now; not sure if there's an official issue logged somewhere.
Rowan Davies
@rowandavies
Apr 28 2015 02:35
@shmookey Yeah, I did consider whether ConnMan was appropriate. Ultimately, I decided it was at least as appropriate as CustomerExecution.
Stephan Hoermann
@stephanh
Apr 28 2015 04:29
FYI I'm planning on working from home tomorrow.
Luke Williams
@shmookey
Apr 28 2015 04:44
if we're giving squoopExport a typedpipe why would we care about the format of the source data?
Vineeth Varghese
@vineethvarghese
Apr 28 2015 04:46
What to mean by format?
Luke Williams
@shmookey
Apr 28 2015 04:47
stephan was asking about parquet support
Vineeth Varghese
@vineethvarghese
Apr 28 2015 04:48
Sqoop doesn't understand parquet
but if your typed pipe is backed by a parquet src...
Luke Williams
@shmookey
Apr 28 2015 04:50
does it not still iterate over whole records?
Stephan Hoermann
@stephanh
Apr 28 2015 04:51
Sqoop export should take a type pipe of something that is a ThriftStruct or Product since that is how we represent the data.
Luke Williams
@shmookey
Apr 28 2015 04:51
do pipes from parquet sources have some other type?
Vineeth Varghese
@vineethvarghese
Apr 28 2015 04:52
but the ThriftStructure is lost at Sqoop while exporting
I guess that doesn't matter
Vineeth Varghese
@vineethvarghese
Apr 28 2015 05:01
@shmookey missed your msg. There is no difference. It is still a typed pipe
Stephan Hoermann
@stephanh
Apr 28 2015 06:10
The file format and wether it is parquet, etc. doesn't matter in this case. Within loads we work on TypedPipe[A <: ThriftStruct] so the sqoop export function you write needs to work with that. However, sqoop is not integrated to scalding so you will have to write the data out in a format that sqoop understands and then run sqoop export on it.
Luke Williams
@shmookey
Apr 28 2015 06:22
yeah, using typedpsv at the moment
Conrad Parker
@kfish
Apr 28 2015 23:31
stand up?