These are chat archives for nextflow-io/nextflow

20th
Feb 2018
mgjmg
@mgjmg
Feb 20 2018 07:40
New to nxtflw; confused as to docker ENTRYPOINT and CMD requirements for single use images; inclined towards executable containers but see this is deprecated; what is recommended/required from a container with executable_x for a nextflow script to run executable_x <args>? (new to containers, too, so on steep learning curve). Thanx
Paolo Di Tommaso
@pditommaso
Feb 20 2018 08:39
there's no a general consensus about that, my preference is not to use any ENTRYPOINT/CMD at all
NF in any case overrides the container ENTRYPOINT when launch a container execution
mgjmg
@mgjmg
Feb 20 2018 08:40
thanks and good morning; in which case how is nxtflw accessing the functionality within the container?
(NF overiride is nice feature .. means I can set up with entrypoint for testing and not have to remove it for use ... but still not understanding how NF runs whatecer is in the container
Paolo Di Tommaso
@pditommaso
Feb 20 2018 08:42
docker run <image> <command to run>
for details see the .command.run file generated by NF in the task dir when using containers
mgjmg
@mgjmg
Feb 20 2018 08:45
?? that is the code NF uses internally when it sees e.g. exe param_1?
Paolo Di Tommaso
@pditommaso
Feb 20 2018 08:46
not understanding
mgjmg
@mgjmg
Feb 20 2018 08:51
https://github.com/SciLifeLab/Sarek/blob/master/main.nf line192: fastqc -q ${fastqFile1} ${fastqFile2}, where fastqc is an image being passed args '-q fq1 fq2'; or do I misundertand (quite possible)
Paolo Di Tommaso
@pditommaso
Feb 20 2018 08:52
the task command is saved to a script named .command.sh
then it's executed as docker run <fastqc image> .command.sh
with some other tricks
mgjmg
@mgjmg
Feb 20 2018 08:57
OK, tricks it is -- do you have time now to juggle with my incomprehension, spell a few things out for me?
Paolo Di Tommaso
@pditommaso
Feb 20 2018 08:58
depends exactly what do want to know
my suggestion, run a NF pipeline, look at the task launcher and then ask on some specific doubt
mgjmg
@mgjmg
Feb 20 2018 09:02
good suggestion; I as yet have none - it there a small exemple pipeline you could recommend (CAW looks great but a tad complex for first steps)?
Paolo Di Tommaso
@pditommaso
Feb 20 2018 09:03
yep
mgjmg
@mgjmg
Feb 20 2018 09:13
ok, thank you; I'll be back ... (-:
mgjmg
@mgjmg
Feb 20 2018 12:08
so do I undertand right that you put the whole of the NF into the image; you are not using images which are single-component SOLID design?
Alexander Peltzer
@apeltzer
Feb 20 2018 12:21
If I understand your question correctly, you mean that the images contain everything required for a workflow?
Thats actually up to you, you can have different containers per process or a single container for each entire workflow (with multiple processes). There are some pros and cons both approaches have
mgjmg
@mgjmg
Feb 20 2018 12:25
No I am asking -- as very new NF would be user -- for you to confirm my impression that the image run by the rnaseqnf project includes everything; that is that the script does not run as an NF script calling in docker imaes for specific tools, but rather that the NF scrupt itself is also within the image. Am I correct (mosly from looking at image using image inspect)
Simone Baffelli
@baffelli
Feb 20 2018 12:36
Hello. Can you confirm that moving the content of the workDir somewhere else (and changing the corresponding path in the config file) changes the hash?
I was hoping to be able to use -resume with the new copy of the workdir, but it seems not to be the case
Alexander Peltzer
@apeltzer
Feb 20 2018 12:38
@mgjmg : No, nextflow calls the docker container with the required dependencies. These are inside the image, whereas nextflow is run on the host in the example of the project you specified. Typically its good practice to run nextflow on the "bare hardware" and dependencies inside containers to be able to reproduce things
if you do something like nextflow run nextflow-io/rnaseq-nf -with-docker that means the nextflow workflow pulls a docker container and executes all individual processes in that container (or as specified in the respective configuration)
Paolo Di Tommaso
@pditommaso
Feb 20 2018 12:40
@baffelli yes it does
Simone Baffelli
@baffelli
Feb 20 2018 12:45
:crying_cat_face:
mgjmg
@mgjmg
Feb 20 2018 12:51
@apeltzer ; OK thank you so that means I don't understand how the NF calls the docker container executables; Dockerfile does various installs -- so the exe are there -- but there is no entrypoint no cmd -- so what is mechanism for the main.nf process fastqc call 'fastqc -o fastqc_${sample_id}_logs -f fastq -q ${reads}' ? that 'fastq' is an image reference (yes?) ; I am not familiar with how to call images like this; can the run-time container be retained (I guess it is run with --rm?) so I can explore this?
Paolo Di Tommaso
@pditommaso
Feb 20 2018 12:55
check the .command.run script in each task directory
mgjmg
@mgjmg
Feb 20 2018 14:25
ok - good pointerl thank you; lentamente things fall into place; I'm not seeing where is built the container && how to ensure all needed containers are built (e.g. if each tool was in its own container the project would have more containers)
Paolo Di Tommaso
@pditommaso
Feb 20 2018 14:27
where is built the container
that's not NF role, workflow developer has to provide that
ensure all needed containers are built
as before
mgjmg
@mgjmg
Feb 20 2018 14:37
And once image for each executable is created they are mapped to process name using an expansion of nextflow.config?
Edgar
@edgano
Feb 20 2018 17:36
does anyone know if it's possible to "capture" the exitcode and use it?
For example, if the exitcode is XX I want to ignore the error and continue the execution
Paolo Di Tommaso
@pditommaso
Feb 20 2018 18:01
foo=$?
more crap pipelines, eh ..
Edgar
@edgano
Feb 20 2018 18:03
hahaha it's not crap pipelines... is crap SW... mBed 'explotes' with big files but it produces the output I expected....
Paolo Di Tommaso
@pditommaso
Feb 20 2018 18:05
you need to use set +ebefore the failing command to capture the error code
Edgar
@edgano
Feb 20 2018 18:05
thanks! :smile:
Paolo Di Tommaso
@pditommaso
Feb 20 2018 18:07
:+1:
Michael L Heuer
@heuermh
Feb 20 2018 19:42
Question re: Docker vs. Singularity, if I have it right, with docker run environment variables are passed in via --env arguments whereas with singularity exec they are set as environment variables prefixed with SINGULARITYENV_?
Paolo Di Tommaso
@pditommaso
Feb 20 2018 21:51
yes, this is a singularity of singularity ..
Michael L Heuer
@heuermh
Feb 20 2018 22:07
Thanks for the confirmation. I had to write something similar to your container package, couldn't use it directly because of incompatibility with Apache license
Vanessasaurus
@vsoch
Feb 20 2018 22:43
@heuermh you can also just create a Dockerfile, and have most of the environment exported in. But if you have runtime determined, you would need to take the custom approach. If it's a matter of multiple entrypoints, you can try out the scientific filesystem to define different environments in the same container (run like different contexts!)
Vanessasaurus
@vsoch
Feb 20 2018 22:54
@mgjmg I might be able to help with multiple entrypoints, I had a lot of trouble with nf too not in running stuffs but in finding files. I have been trying out snakemake with containers and it seems to be a little more intituive https://github.com/sci-f/snakemake.scif but if you get nf working I'd like to know how you did, so please share!
Vanessasaurus
@vsoch
Feb 20 2018 23:05
I really like gitter, it's fun to talk to you guys on here <3
Michael L Heuer
@heuermh
Feb 20 2018 23:30
Yep, the longer you are here, the more beer you will owe @pditommaso when you finally meet him :)
:beers: