These are chat archives for nextflow-io/nextflow

29th
Apr 2019
longkaizheng
@longkaizheng
Apr 29 01:08
Hi All! I am running the nextflow, but the nextflow would stop automatically , and the tasks I delivered would be running with no default.
The logs are as follows:
DEBUG nextflow.executor.GridTaskHandler - Failed to get exit status for process TaskHandler[jobId: 169332; id: 1; name:
map (TI1706140112LD02); status: RUNNING; exit: -; error: -; workDir: /share/udata/cuiheyang/nextflow_GATK/work/01/7a807dcca531d0ba591132a644e9e3 started: 15
56489349535; exited: -; ] -- exitStatusReadTimeoutMillis: 270000; delta: 270002
Jonathan Manning
@pinin4fjords
Apr 29 07:07
Which version are you using @longkaizheng ? I had this in nextflow-io/nextflow#1045, solved with latest version.
Hugues Fontenelle
@huguesfontenelle
Apr 29 09:03
Good morning!
@pditommaso , Do you have any plan to resuscitate support for DNAnexus? We could make good use of it ;-)
I understand that one reason for deprecation was that DNAnexus did not support Docker, but that's not the case anymore!
Paolo Di Tommaso
@pditommaso
Apr 29 09:12
hardly with nextflow, we can discuss some kind of support for DNAnexus through https://seqera.io
Hugues Fontenelle
@huguesfontenelle
Apr 29 09:24
Thanks. I'll shoot you an email to seqera some time this week!
Paolo Di Tommaso
@pditommaso
Apr 29 09:24
+1
longkaizheng
@longkaizheng
Apr 29 12:11
@pinin4fjords Thank you for your help! I will have a try!
Mazen Mahdi
@Shamanga_13_twitter
Apr 29 13:28
hello all,
I've been struggling with piping the output of a docker container to another.
Here is a sample code that shows conceptually what I am trying to do:
#!/usr/bin/env nextflow

myFileChannel = Channel.fromPath('/some/input/images')

process process_1 {

cpus 1
maxForks 1


input:
file x from myFileChannel

output:
file "output_image" into output_process_1

containerOptions "--volume /path/to/input/:/input --volume /path/to/output:/output"

"""
cd /
./executable_1 /input/$x -output /output/output_image
"""
}


process process_2 {

cpus 1
maxForks 1

input:
file x from output_process_1

containerOptions "--volume /path/to/input/:/input --volume /path/to/output:/output"

"""
cd /
./executable_2 /input/$x -output /output/output_image
"""
}
any ideas?
Hugues Fontenelle
@huguesfontenelle
Apr 29 13:32
What are you trying to accomplish and why?
Outputing files into channels should do all the heavy work, without any additional mounting.
Mazen Mahdi
@Shamanga_13_twitter
Apr 29 13:36
I am trying to build a pipeline that uses several docker images as processing blocks.
The input of each docker image depends on the output of the previous one.
In the simple example above, I am passing an input image to the first container ( process_1). This container outputs an image, that I need as input to process_2
micans
@micans
Apr 29 13:36
cd'ing out of the work directory is against the grain of nextflow, I have no experience with that. I imagine hooking up files with channels becomes difficult or impossible. Change the output option to something like -output output_image. Is there a reason you change directory to /?
Mazen Mahdi
@Shamanga_13_twitter
Apr 29 13:37
@micans the executables are in the / directory
micans
@micans
Apr 29 13:38
then change ./executable_2 to /executable_2?
Hugues Fontenelle
@huguesfontenelle
Apr 29 13:38
@Shamanga_13_twitter What do you mean by "image"?
You've got a nf pipeline, running in docker, that produces docker images?
Mazen Mahdi
@Shamanga_13_twitter
Apr 29 13:41
@huguesfontenelle , the docker image takes as input a nifti file and outputs a nifti file
Hugues Fontenelle
@huguesfontenelle
Apr 29 13:42
OK. What is it that wasn't working with simply passing the NIFTI file in the channel?
Mazen Mahdi
@Shamanga_13_twitter
Apr 29 13:44
@micans this is just a quick draft of the concept
Hugues Fontenelle
@huguesfontenelle
Apr 29 13:46
#!/usr/bin/env nextflow
myFileChannel = Channel.fromPath('/path/to/input.nifti')

process process_1 {
publishDir "output/p1"

input:
file x from myFileChannel

output:
file "output1.nifti" into output_process_1

"""
/path/to/executable_1 $x -output output1.nifti
"""
}

process process_2 {
publishDir "output/p2"

input:
file x from output_process_1

output:
file "output2.nifti"

"""
/path/to/executable_2 $x -output output2.nifti
"""
}
micans
@micans
Apr 29 13:46
@Shamanga_13_twitter that makes it harder to know what the actual problem is then.
Hugues Fontenelle
@huguesfontenelle
Apr 29 13:48
The script I drafted would be the standard approach, as far as I know and understand so far ;-)
micans
@micans
Apr 29 13:48
seconded :-)
Mazen Mahdi
@Shamanga_13_twitter
Apr 29 13:49
@huguesfontenelle, i'll try this and let you know.
thanks guys
Hugues Fontenelle
@huguesfontenelle
Apr 29 13:49
Nice!
Mazen Mahdi
@Shamanga_13_twitter
Apr 29 15:27
@huguesfontenelle, thanks a lot. It does exactly as I needed
Hugues Fontenelle
@huguesfontenelle
Apr 29 16:46
You're welcome, also thanks for letting me know. Check out https://github.com/nextflow-io/awesome-nextflow and https://github.com/nf-core for inspiration about how to build pipelines. I'm learning a lot by reading those :-)
Tobias "Tobi" Schraink
@tobsecret
Apr 29 21:10
Anyone ever had a problem where executables cannot be found by python, even though they are in the bin folder?
Tobias "Tobi" Schraink
@tobsecret
Apr 29 21:15
Hmmm, maybe I'm not really understanding how it's supposed to work - using the workaround python ${baseDir}/executable.py
Laurence E. Bernstein
@lebernstein
Apr 29 22:15
I use full paths to my python scripts so I never noticed. :)