These are chat archives for nextflow-io/nextflow

17th
May 2016
Jason Byars
@jbyars
May 17 2016 15:52
should storeDir work with S3 buckets or is that only for publishDir?
Paolo Di Tommaso
@pditommaso
May 17 2016 15:53
oh, good question
I guess S3 not works for storeDir but I'm not 100% sure
however I discourage to use storeDir, in most cases publishDir works better
Jason Byars
@jbyars
May 17 2016 15:58
storeDir does not appear to work with S3. I would prefer to use publishDir, but if I use publishDir, how do I get an equivalent when declaration for results in buckets?
Paolo Di Tommaso
@pditommaso
May 17 2016 15:59
what's your use case?
Jason Byars
@jbyars
May 17 2016 16:06
Results go in buckets, if I run a workflow again with more files in the input bucket, I don't want to reanalyze what I've already processed.
My data trickles in
Does this make sense?
Paolo Di Tommaso
@pditommaso
May 17 2016 16:11
well, I think you can manage to implement it by using a publishDir rule and skipping the process execution when a marker file in the published dir already exist
for example:
Jason Byars
@jbyars
May 17 2016 16:11
Yes, that is the idea.
Paolo Di Tommaso
@pditommaso
May 17 2016 16:12
process foo {
  publishDir 's3://bucket/my-data/'
  when:
  !file('s3://bucket/my-data/marker').exists()
  output: 
  file ('marker') ... 

}
it should work
Jason Byars
@jbyars
May 17 2016 16:13
It should, but it didn't when I tried it. Let me go see what typo I made and get back to you.
Jason Byars
@jbyars
May 17 2016 16:25
file() not File(), it works now. Thanks
Jason Byars
@jbyars
May 17 2016 16:34
Have you ever queried a DynamoDB table in groovy? That's where I would really like to put the marker.
Paolo Di Tommaso
@pditommaso
May 17 2016 16:35
Uh, so it's not S3
Never tried, but using the Java api should work
Jason Byars
@jbyars
May 17 2016 16:36
It's S3 right now. Eventually, I want to use spot instances and I don't want to clutter buckets with flag files.
The java api should work. I'm just making sure I'm not over looking a keyword.
Paolo Di Tommaso
@pditommaso
May 17 2016 16:39
Indeed, interesting stack
Jason Byars
@jbyars
May 17 2016 16:40
Great, this clears up enough, I can put an example together.
Mike Smoot
@mes5k
May 17 2016 17:08
@pditommaso I'm wondering if you've considered a process errorStrategy that terminates the current process when there's an error, but waits for any other running processes to complete before terminating the pipeline?
Paolo Di Tommaso
@pditommaso
May 17 2016 17:11
No, but I guess it would be useful to do not waste long running jobs. Right?
Mike Smoot
@mes5k
May 17 2016 17:14
Exactly!
Paolo Di Tommaso
@pditommaso
May 17 2016 17:16
Makes sense. Please open a feature request on GH suggesting a name for it
Mike Smoot
@mes5k
May 17 2016 17:18
Will do, thanks.
Jason Byars
@jbyars
May 17 2016 19:11
I would definitely benefit from that errorStrategy as well. It would be really neat if the failing process could also trigger some onComplete equivalent so you could be notified when a process fails instead of finding out the next morning when everything finishes.
Paolo Di Tommaso
@pditommaso
May 17 2016 19:56
@jbyars It already does it. You need to check the value of workflow.success
workflow.onComplete {
    println "Pipeline completed at: $workflow.complete"
    println "Execution status: ${ workflow.success ? 'OK' : 'failed' }"
}
Jason Byars
@jbyars
May 17 2016 19:56
workflow.success would trigger in the middle of a run, when a single process fails?
Paolo Di Tommaso
@pditommaso
May 17 2016 19:57
ah, you mean at process level not the entire workflow
Jason Byars
@jbyars
May 17 2016 19:58
exactly! Give me my bad news when it happens, not 5 minutes before lab meeting
Paolo Di Tommaso
@pditommaso
May 17 2016 19:58
:)
yes, but unless you set to ignore process errors the pipeline execution stops when a process fail
and if you want to ignore there's no reason to get an error notification
Does make sense?
Jason Byars
@jbyars
May 17 2016 20:00
right, that's why I would like this to be considered for the strategy Mike proposed.
Paolo Di Tommaso
@pditommaso
May 17 2016 20:00
I see
Jason Byars
@jbyars
May 17 2016 20:02
I like the idea of not killing everything when one job fails for long running jobs, but I don't want to ignore errors, or wait until the end to find out about them.
Paolo Di Tommaso
@pditommaso
May 17 2016 20:03
quite clear
Jason Byars
@jbyars
May 17 2016 20:04
the retry approach is great when I know I just need to throw more memory at the problem or any similar straight forward situation. Usually I'm not that lucky
Jason Byars
@jbyars
May 17 2016 20:10
this is really interesting to me because the Slack integration for Jenkins (and most notification systems) just tell me something succeeded or failed. With a mechanic like workflow.onComplete I can dump whatever I want at the end of a run. Add a process.onFail that I can dump everything relevant to the failing task and when my phone goes off I know where to start debugging.
Paolo Di Tommaso
@pditommaso
May 17 2016 20:13
I think it could be possible to add some notification handler like workflow.onProcessComplete, workflow.onProcessFailed
wow .. 198 stargazers on GH
Who is next? :)
Jason Byars
@jbyars
May 17 2016 20:38
Congratulations! Shall I add a feature request for workflow.onProcessComplete, workflow.onProcessFailed?
Paolo Di Tommaso
@pditommaso
May 17 2016 20:39
Yes, please
Mike Smoot
@mes5k
May 17 2016 20:52
@pditommaso I'm running into a problem where a process fails because of a missing expected output file even though that file appears to exist. I'm wondering if this might be a race condition. Any ideas on where I might look or debug what nextflow thinks it's seeing?
Paolo Di Tommaso
@pditommaso
May 17 2016 20:53
are you running it over a shared file system ?
Mike Smoot
@mes5k
May 17 2016 20:58
It's running in a docker container and it's pulling some of the original data from an NFS share, but the files in question should all be on the local disk (i.e. in the work dir)
I can paste in the process if that helps clarify things.
Paolo Di Tommaso
@pditommaso
May 17 2016 20:59
ok
Mike Smoot
@mes5k
May 17 2016 21:00
process calculateGenomeSizes {
    container 'dockreg-dev01.awsv.l.synthgeno.global/tools/ucsc:v326'
    errorStrategy 'retry'

    input:
    set gid, file(fasta) from gid_rf_calc_genome

    output:
    set gid, stdout, "${gid}.genomeSizes.txt" into genome_id__size__file

    script:
    """
    faCount ${fasta} | grep ^total | sed 's/\t/ /g' | cut -f2 -d ' ' | tee ${gid}.genomeSize.txt
    """
}
It complains that ${gid}.genomeSizes.txt is missing even though I see it in work.
And the 'retry' was in there to handle this problem, which isn't what I'd normally like.
Paolo Di Tommaso
@pditommaso
May 17 2016 21:02
it sounds very strange
it's not deterministic?
Mike Smoot
@mes5k
May 17 2016 21:03
In my example I've got 8 genomes and it seems to fail on a different one each time.
Paolo Di Tommaso
@pditommaso
May 17 2016 21:05
this is not using a cluster, right? just a local execution
Mike Smoot
@mes5k
May 17 2016 21:05
The files do get owner/user as root, which I'm assuming is an artifact of running in docker, but if I add a chmod to the command above nothing seems to change.
Yes, running locally.
Paolo Di Tommaso
@pditommaso
May 17 2016 21:06
very strange I seen something similar when using a cluster engine
but never on a local storage
I'm wondering if could be somehow related to docker
Mike Smoot
@mes5k
May 17 2016 21:08
Yeah, I'm wondering that too. I'll try installing faCount locally and see if that helps.
Paolo Di Tommaso
@pditommaso
May 17 2016 21:09
or some command in your script not closing properly the file
tee could be a good canditate for such a problem
I would try to replace tee with a simple redirect, at least to test it
Mike Smoot
@mes5k
May 17 2016 21:11
will try that next
Jason Byars
@jbyars
May 17 2016 21:19
@pditommaso This sounds somewhat like the docker 1.9.x version issue you helped me with a couple weeks ago.
Paolo Di Tommaso
@pditommaso
May 17 2016 21:19
oh, it could
@mes5k which docker version are you using?
Mike Smoot
@mes5k
May 17 2016 21:23
1.9.1!
Paolo Di Tommaso
@pditommaso
May 17 2016 21:23
got it!
Mike Smoot
@mes5k
May 17 2016 21:24
Although I'm now seeing the same behavior without docker...
Paolo Di Tommaso
@pditommaso
May 17 2016 21:25
really? have you tried to remove the tee?
Mike Smoot
@mes5k
May 17 2016 21:25
Yup, removed the tee, removed all pipes ( now there are a bunch of temp files)
What was the docker problem you identified with 1.9.x?
Paolo Di Tommaso
@pditommaso
May 17 2016 21:26
no docker, no pipes and the same problem?
Mike Smoot
@mes5k
May 17 2016 21:27
Yeah, and the command is clearly running and showing the correct result. Strange.
Jason Byars
@jbyars
May 17 2016 21:28
docker 1.9.x + nfs or any shared filesystem seems to have weird behavior. You see the files but can't access them, output files are empty, nextflow can't see the output files, etc.
Mike Smoot
@mes5k
May 17 2016 21:29
Well, that certainly matches the symptoms. Updating (or downgrading) the version of docker helped?
Jason Byars
@jbyars
May 17 2016 21:31
docker 1.10.x and newer seem to be fine. I ran into this on Ubuntu 14.04 and Amazon Linux.
Mike Smoot
@mes5k
May 17 2016 21:31
I'll see about upgrading. I'm running this on Centos 7.2.
This is only a small part of the pipeline and something I can work around for now. I'll keep banging on this and keep you posted on what I find. Thanks for the help!
Jason Byars
@jbyars
May 17 2016 21:33
if you're having the problem without docker... I'd like to know the punchline when you figure it out.
Paolo Di Tommaso
@pditommaso
May 17 2016 21:33
+1
Mike Smoot
@mes5k
May 17 2016 21:34
I wonder if it's docker being used in a different part of the pipeline? The prior process is still using docker.