These are chat archives for nextflow-io/nextflow

4th
Apr 2016
Matthieu Foll
@mfoll
Apr 04 2016 14:50
This message was deleted
This message was deleted
Matthieu Foll
@mfoll
Apr 04 2016 14:57
This message was deleted
Michael L Heuer
@heuermh
Apr 04 2016 15:28
@pditommaso might you be making it to BOSC or ISMB this year?
Paolo Di Tommaso
@pditommaso
Apr 04 2016 15:56
@heuermh Not sure, up to day more no than yes. Are you going ?
Michael L Heuer
@heuermh
Apr 04 2016 15:58
Yes. The BOSC abstract submission deadline was extended to today, so I was going to ask you to submit one. :)
Paolo Di Tommaso
@pditommaso
Apr 04 2016 15:58
:)
I've not submitted this year, have you?
Michael L Heuer
@heuermh
Apr 04 2016 16:00
we've submitted abstracts for our projects ADAM and Mango
still am not sure exactly how to call Spark pipelines from nextflow, but will get it figured out some day
ping me if you decide to go, it'd be fun to meet up
Paolo Di Tommaso
@pditommaso
Apr 04 2016 16:02
yep, maybe we could arrange an hacking session at the Codefest
Michael L Heuer
@heuermh
Apr 04 2016 16:02
that would be very cool
Paolo Di Tommaso
@pditommaso
Apr 04 2016 16:04
ok, I will let you know
Michael L Heuer
@heuermh
Apr 04 2016 16:05
we've been collaborating with UCSC and haven't exactly figured out the best way to call Spark from Toil pipelines either
Paolo Di Tommaso
@pditommaso
Apr 04 2016 16:06
that looks more difficult since Toil is written in Python
Michael L Heuer
@heuermh
Apr 04 2016 16:12
it works, we're still working on handling dropped spot instances and such
Paolo Di Tommaso
@pditommaso
Apr 04 2016 16:13
what was the rationale on choosing Toil? Collaboration driven?
Michael L Heuer
@heuermh
Apr 04 2016 16:14
Yup.
Toil is adding support for SLURM, and CWL, and possibly running directly on YARN, so there's some overlapping technology; I hope to learn from our approach(es) with Toil what might work best with Nextflow
Paolo Di Tommaso
@pditommaso
Apr 04 2016 16:18
I'm still thinking that nextflow could run natively on a spark cluster, thus having a much better integration also for ADAM tasks
Michael L Heuer
@heuermh
Apr 04 2016 16:21
Hmm, hadn't thought of that
Paolo Di Tommaso
@pditommaso
Apr 04 2016 16:21
:)
If I come to Orlando we will ;)
Michael L Heuer
@heuermh
Apr 04 2016 16:22
+1
Jason Byars
@jbyars
Apr 04 2016 21:24
are there any restrictions on the workflow.onComplete handler that would prevent a curl webhook from working?
Paolo Di Tommaso
@pditommaso
Apr 04 2016 21:25
I'm not aware of that. Are you experiencing any problem?
Jason Byars
@jbyars
Apr 04 2016 21:28
I'm 99% sure I've just got a mistake escaping characters. When I try to send a slack notification it's just not working.
but if I print the command string out and paste it on the command line it works.
Paolo Di Tommaso
@pditommaso
Apr 04 2016 21:30
weird
you may want to try a native HttpUrlConnection
Jason Byars
@jbyars
Apr 04 2016 21:31
that's worth a try
Jason Byars
@jbyars
Apr 04 2016 21:42
where is stdout/stderr routed for workflow.onComplete
Paolo Di Tommaso
@pditommaso
Apr 04 2016 21:43
onComplete executes groovy/java code, thus there's no stdout/stderr
Jason Byars
@jbyars
Apr 04 2016 21:48
right, sorry I mean in onComplete, if I do a log.info "some command".execute().text normally, I can print the output and get a quick hint.
Paolo Di Tommaso
@pditommaso
Apr 04 2016 21:51
that should print the stdout. is it not working ?
Jason Byars
@jbyars
Apr 04 2016 21:54
exactly, I get nothing from curl. Other commands seem to work fine. I better see if there's some weird quirks for curl -v. Maybe it outputs to stderr
Paolo Di Tommaso
@pditommaso
Apr 04 2016 21:55
um, I think you need to wait for the completion of the process before read the text
something like
Jason Byars
@jbyars
Apr 04 2016 21:55
ahh that's probably it
Paolo Di Tommaso
@pditommaso
Apr 04 2016 21:55
def proc = "<command>" .execute()
proc.waitFor()
println proc.text
This message was deleted
Jason Byars
@jbyars
Apr 04 2016 21:58
that was it, I forgot to wait for the process to finish.
thanks
Paolo Di Tommaso
@pditommaso
Apr 04 2016 21:59
welcome
Jason Byars
@jbyars
Apr 04 2016 22:45
it's working. I actually had to use the list version [].execute() to get around all of the quoting issues.
Paolo Di Tommaso
@pditommaso
Apr 04 2016 22:46
yes, much better