These are chat archives for nextflow-io/nextflow

13th
Apr 2017
Félix C. Morency
@fmorency
Apr 13 2017 14:01 UTC
how does NF handles bin/ and templates/ when using -with-singularity?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 14:01 UTC
the same way as Docker
Félix C. Morency
@fmorency
Apr 13 2017 14:02 UTC
which is? :P
Paolo Di Tommaso
@pditommaso
Apr 13 2017 14:02 UTC
templates are not a problem at all, because that are just text scripts parsed by NF
the bin/ folder is added in the (container) PATH
Félix C. Morency
@fmorency
Apr 13 2017 14:03 UTC
Oh cool!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 14:03 UTC
;)
Félix C. Morency
@fmorency
Apr 13 2017 14:04 UTC
Thanks.
Paolo Di Tommaso
@pditommaso
Apr 13 2017 14:04 UTC
:+1:
Karin Lagesen
@karinlag
Apr 13 2017 15:21 UTC
...what do I do when I need to have quotation marks inside of the script section?
example (which really doesn't work)
cat ${pair_id}_ariba/mlst_report.tsv | awk '\{print ${pair_id}, $0\}' >> ${pair_id}_mlst_report.tsv
Paolo Di Tommaso
@pditommaso
Apr 13 2017 15:23 UTC
you need to escape the \ and $
thus
cat ${pair_id}_ariba/mlst_report.tsv | awk '\\{print ${pair_id}, \$0\\}' >> ${pair_id}_mlst_report.tsv
Karin Lagesen
@karinlag
Apr 13 2017 15:24 UTC
ahhhh
thanks!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 15:24 UTC
I know a but annoying
otherwise you can create an external bash script
Karin Lagesen
@karinlag
Apr 13 2017 15:25 UTC
have thought of it
but will try this way a bit more
Paolo Di Tommaso
@pditommaso
Apr 13 2017 15:25 UTC
ok
Karin Lagesen
@karinlag
Apr 13 2017 15:25 UTC
but thanks :)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 15:25 UTC
welcome!
Félix C. Morency
@fmorency
Apr 13 2017 16:11 UTC
Anyone had Permission deniedissue when using a .sh located in pipeline/bin with -with-singularity?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 16:14 UTC
um, you gave exec permission, right?
Félix C. Morency
@fmorency
Apr 13 2017 16:14 UTC
The permissions from inside the container are -rwxrwxr-x.
Paolo Di Tommaso
@pditommaso
Apr 13 2017 16:15 UTC
So why it produces a Permission denied? Singularity issue ?
Félix C. Morency
@fmorency
Apr 13 2017 16:16 UTC
This is what I am trying to clarify
Paolo Di Tommaso
@pditommaso
Apr 13 2017 16:17 UTC
you could just run a singularity exec .. etc to verify it
Félix C. Morency
@fmorency
Apr 13 2017 17:00 UTC
Ah got it
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:00 UTC
what it was ?
Félix C. Morency
@fmorency
Apr 13 2017 17:00 UTC
TIL, the user mount option turns off exec by default
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:02 UTC
never used that option, what is useful for ?
Félix C. Morency
@fmorency
Apr 13 2017 17:02 UTC
Allow regular user to mount entries in fstab
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:02 UTC
I see
but what do you mean turns off exec ?
ah, the execute permissions!
how to make it works if so ?
Félix C. Morency
@fmorency
Apr 13 2017 17:04 UTC
just explicitly pass the exec option to mount
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:04 UTC
um, how ?
Félix C. Morency
@fmorency
Apr 13 2017 17:05 UTC
In the /etc/fstab
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:05 UTC
I see, good to know
Tim Diels
@timdiels
Apr 13 2017 17:44 UTC
Is there a way to access -name {name} in main.nf?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:45 UTC
yes workflow.runName
Karin Lagesen
@karinlag
Apr 13 2017 17:54 UTC
...what would be the best strategy if I would want to dump quite a bit of that metadata to a file when I run a workflow? Do I create a process for that, or...?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:56 UTC
that metadata
well, what metadata ?
Karin Lagesen
@karinlag
Apr 13 2017 18:00 UTC
such as commandline name, run date, commitID etc
i.e. amongst the $workflow metadata mentioned above
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:00 UTC
ah
Félix C. Morency
@fmorency
Apr 13 2017 18:01 UTC
@karinlag I dump them to stdout at the beginning of a run
they are logged in the nextflow log file
Karin Lagesen
@karinlag
Apr 13 2017 18:01 UTC
I prepare all the results in a publishDir directory
and I'd ideally like to have this kind of metadata stuffed in a file in that directory
so that my users don't have to do any thinking when running :)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:02 UTC
you can do easily that
file('/publish/dir/path/metadata').text = """
 commitId: ${workflow. commitId}
 ... 
"""
Karin Lagesen
@karinlag
Apr 13 2017 18:04 UTC
thanks!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:04 UTC
would be nice to have a build-in method to do that, something like
Tim Diels
@timdiels
Apr 13 2017 18:05 UTC
@pditommaso Thanks
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:05 UTC
workflow.saveMetadata( path )
@timdiels welcome
Karin Lagesen
@karinlag
Apr 13 2017 18:05 UTC
yes indeed!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:06 UTC
if you can help me to not forget it opening a feature request on GH, I may try to do it
:)
Evan Floden
@evanfloden
Apr 13 2017 18:06 UTC
:+1: Nice idea
Mike Smoot
@mes5k
Apr 13 2017 18:11 UTC
+1 here too
Alessia
@alesssia
Apr 13 2017 18:50 UTC
Hello, this is surely a stupid question :) I am trying to run a nextflow pipeline using a Docker container and the option -with-docker mydocker, but how do I mount the disk? What I would usually do with docker run --volume ~/myfolder:myfolder?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:51 UTC
ciao @alesssia not a stupid question at all
NF mounts automatically the host paths required to access all inputs files
so you don't have to!
Alessia
@alesssia
Apr 13 2017 18:53 UTC
mmm, so the error I receive saying that the file does not exist does not depend on this...
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:54 UTC
I have no idea w/o showing the error message
Alessia
@alesssia
Apr 13 2017 18:58 UTC
I am running a process that uses a bash script. First line creates a log file, as log=\"$workingpath/.log.c2\" then redirect a message on it (echo \"something\" > \$log and what I get is .command.sh: line 8: ./.log.c2 : No such file or directory (but if I run the .command.sh script it works...)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:01 UTC
$workingpath is supposed to be the local path ie. . ?
Alessia
@alesssia
Apr 13 2017 19:01 UTC
I tried with both local and absolute path
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:02 UTC
the NF way is to work in the tasks local dir
so you could just do
log=".log.c2"
echo "something" > \$log
it should work in this way
Alessia
@alesssia
Apr 13 2017 19:04 UTC
That does not work on our HPC (but notably it works with the absolute path), trying to have NF working an all system (that is: testing it on my machine now)
the NF script work fine on HPC (without docker) no way of having it running on my laptop (MacBook Pro)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:05 UTC
the NF script work fine on HPC (without docker) no way of having it running on my laptop (MacBook Pro)
because I suspect you have used absolute path somewhere
Alessia
@alesssia
Apr 13 2017 19:06 UTC
Nein :)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:06 UTC
but notably it works with the absolute path
?
Alessia
@alesssia
Apr 13 2017 19:06 UTC
it is a command line parameter
that is then added to all the filepath (--outdir=/something/somewhere/)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:07 UTC
um
Alessia
@alesssia
Apr 13 2017 19:07 UTC
workingpath = params.outdir + "/" + params.prefix
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:07 UTC
ah!
any path/file that you use in your process script it must be properly declared as an input
otherwise it may not work with some executor or when using containers
Alessia
@alesssia
Apr 13 2017 19:10 UTC
so doing nextflow run pipeline.nf --outdir /something/somewhere/ does not mean declaring it?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:11 UTC
nope, that's just passing a parameter
I mean something like
Alessia
@alesssia
Apr 13 2017 19:11 UTC
workingpath = params.outdir + "/" + params.prefix
workingdir = file(workingpath)
if( !workingdir.exists() ) {
    if( !workingdir.mkdirs() )     {
        exit 1, "Cannot create working directory: $workingpath"
    } 
}
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:11 UTC
process foo {
  input: 
  file x from bar_ch

  """
  cat $x
  """
}
Alessia
@alesssia
Apr 13 2017 19:12 UTC
I can't use workingdir that is declared as file(workingpath)?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:12 UTC
I mean process input, as showed above
you should never use an absolute path in a process script
Alessia
@alesssia
Apr 13 2017 19:13 UTC
mmm, so I need to create a channel for each process just to set up the working dir (I can' reuse the same channel, can I?)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:13 UTC
nope
you don't need to setup the work dir, this is exactly the thing that makes NF different from the other tools
your tasks need to read/write in (their unique) work dir that is assigned by NF
Alessia
@alesssia
Apr 13 2017 19:15 UTC
no, wait a second, not sure I have understood. I have 11 process, each of them creates multiple files. Shoud I set up a channel for each of these outputs?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:16 UTC
each for each output yes, but an output can capture one or more file or even a directory
in practice usually you have an output for each process, even if the task produce multiple files
Alessia
@alesssia
Apr 13 2017 19:18 UTC
Ok, I need to think about this. Just wondering why it works fine on HPC :confused:
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:20 UTC
because, there you force to use the share dir, but I guess if you try to resume the execution with the -resume option won't work
moreover when you use docker
NF needs to mount the file system using the input/output information
thus it won't work anymore
Alessia
@alesssia
Apr 13 2017 19:22 UTC
I see, ok. Need to do a few changes then!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:22 UTC
to resume, if you want to make sure to have your pipeline portable taking advantage of all NF features
you need to make sure that each task works in its own
seems difficult, but you will realise that simplifies a lot of things
at least I hope so :)
here there's a short tutorial to NF + Docker, you may want to give a look
Alessia
@alesssia
Apr 13 2017 19:24 UTC
I looked at it :)
Evan Floden
@evanfloden
Apr 13 2017 19:24 UTC
Here is my basic take on it, not sure it helps but:
  • Think of each time a process is executed as a task.
  • Each task lives in it own working directory (which is the the workfolder).
  • With NF, you simply specify which files should go into that that work dir (inputs)
  • and which files you want to use later on (outputs).
  • The channels simply move the inputs/outputs around.
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:24 UTC
oops! :D
Alessia
@alesssia
Apr 13 2017 19:24 UTC
I think I need a few changes (and I knew that it was a stupid question ;) )
Evan Floden
@evanfloden
Apr 13 2017 19:25 UTC
If you share your code on pastebin I’m sure you will find someone to help you out
Alessia
@alesssia
Apr 13 2017 19:25 UTC
I think that now that I got where is the issue I can try by myself!
Evan Floden
@evanfloden
Apr 13 2017 19:26 UTC
:+1:
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:26 UTC
That's a good approach! ;)
Alessia
@alesssia
Apr 13 2017 19:26 UTC
will keep it as last shore (in few days)!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:26 UTC
ahah
Alessia
@alesssia
Apr 13 2017 19:28 UTC
Thank you!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:29 UTC
Have fun
Alessia
@alesssia
Apr 13 2017 19:32 UTC
:D
amacbride
@amacbride
Apr 13 2017 21:01 UTC

@pditommaso Is it possible to access the SLURM environment variables from within a template sh script? In particular, I'd like to see the value of SLURM_JOB_NODELIST.

I tried the following, but the environment variable is blank, so I'm assuming I escaped it incorrectly?

#!/bin/bash

echo ""
echo "----"
echo "${task.process}"
echo "execution host: `hostname -s`"
echo "slurm host: \$SLURM_JOB_NODELIST"
It's in the .command.log file, so I figured it must be possible to access it from the template script somehow.
SLURMD_NODENAME=marin
SLURM_GTIDS=0
SLURM_JOB_CPUS_PER_NODE=1
SLURM_JOB_ID=49559
SLURM_JOBID=49559
SLURM_JOB_NAME=nf-envtest
SLURM_JOB_NODELIST=marin
SLURM_JOB_NUM_NODES=1
SLURM_LOCALID=0
SLURM_NNODES=1
SLURM_NODE_ALIASES=(null)
SLURM_NODEID=0
SLURM_NODELIST=marin
Paolo Di Tommaso
@pditommaso
Apr 13 2017 21:32 UTC
yes, like any other variable
I don't see why it should not
amacbride
@amacbride
Apr 13 2017 23:59 UTC
Here's the output of that first script (this was run with 0.24.1):
----
envtest
execution host: marin
slurm host: 
+ sleep 30