These are chat archives for nextflow-io/nextflow

13th
Apr 2017
Félix C. Morency
@fmorency
Apr 13 2017 14:01
how does NF handles bin/ and templates/ when using -with-singularity?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 14:01
the same way as Docker
Félix C. Morency
@fmorency
Apr 13 2017 14:02
which is? :P
Paolo Di Tommaso
@pditommaso
Apr 13 2017 14:02
templates are not a problem at all, because that are just text scripts parsed by NF
the bin/ folder is added in the (container) PATH
Félix C. Morency
@fmorency
Apr 13 2017 14:03
Oh cool!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 14:03
;)
Félix C. Morency
@fmorency
Apr 13 2017 14:04
Thanks.
Paolo Di Tommaso
@pditommaso
Apr 13 2017 14:04
:+1:
Karin Lagesen
@karinlag
Apr 13 2017 15:21
...what do I do when I need to have quotation marks inside of the script section?
example (which really doesn't work)
cat ${pair_id}_ariba/mlst_report.tsv | awk '\{print ${pair_id}, $0\}' >> ${pair_id}_mlst_report.tsv
Paolo Di Tommaso
@pditommaso
Apr 13 2017 15:23
you need to escape the \ and $
thus
cat ${pair_id}_ariba/mlst_report.tsv | awk '\\{print ${pair_id}, \$0\\}' >> ${pair_id}_mlst_report.tsv
Karin Lagesen
@karinlag
Apr 13 2017 15:24
ahhhh
thanks!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 15:24
I know a but annoying
otherwise you can create an external bash script
Karin Lagesen
@karinlag
Apr 13 2017 15:25
have thought of it
but will try this way a bit more
Paolo Di Tommaso
@pditommaso
Apr 13 2017 15:25
ok
Karin Lagesen
@karinlag
Apr 13 2017 15:25
but thanks :)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 15:25
welcome!
Félix C. Morency
@fmorency
Apr 13 2017 16:11
Anyone had Permission deniedissue when using a .sh located in pipeline/bin with -with-singularity?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 16:14
um, you gave exec permission, right?
Félix C. Morency
@fmorency
Apr 13 2017 16:14
The permissions from inside the container are -rwxrwxr-x.
Paolo Di Tommaso
@pditommaso
Apr 13 2017 16:15
So why it produces a Permission denied? Singularity issue ?
Félix C. Morency
@fmorency
Apr 13 2017 16:16
This is what I am trying to clarify
Paolo Di Tommaso
@pditommaso
Apr 13 2017 16:17
you could just run a singularity exec .. etc to verify it
Félix C. Morency
@fmorency
Apr 13 2017 17:00
Ah got it
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:00
what it was ?
Félix C. Morency
@fmorency
Apr 13 2017 17:00
TIL, the user mount option turns off exec by default
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:02
never used that option, what is useful for ?
Félix C. Morency
@fmorency
Apr 13 2017 17:02
Allow regular user to mount entries in fstab
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:02
I see
but what do you mean turns off exec ?
ah, the execute permissions!
how to make it works if so ?
Félix C. Morency
@fmorency
Apr 13 2017 17:04
just explicitly pass the exec option to mount
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:04
um, how ?
Félix C. Morency
@fmorency
Apr 13 2017 17:05
In the /etc/fstab
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:05
I see, good to know
Tim Diels
@timdiels
Apr 13 2017 17:44
Is there a way to access -name {name} in main.nf?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:45
yes workflow.runName
Karin Lagesen
@karinlag
Apr 13 2017 17:54
...what would be the best strategy if I would want to dump quite a bit of that metadata to a file when I run a workflow? Do I create a process for that, or...?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 17:56
that metadata
well, what metadata ?
Karin Lagesen
@karinlag
Apr 13 2017 18:00
such as commandline name, run date, commitID etc
i.e. amongst the $workflow metadata mentioned above
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:00
ah
Félix C. Morency
@fmorency
Apr 13 2017 18:01
@karinlag I dump them to stdout at the beginning of a run
they are logged in the nextflow log file
Karin Lagesen
@karinlag
Apr 13 2017 18:01
I prepare all the results in a publishDir directory
and I'd ideally like to have this kind of metadata stuffed in a file in that directory
so that my users don't have to do any thinking when running :)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:02
you can do easily that
file('/publish/dir/path/metadata').text = """
 commitId: ${workflow. commitId}
 ... 
"""
Karin Lagesen
@karinlag
Apr 13 2017 18:04
thanks!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:04
would be nice to have a build-in method to do that, something like
Tim Diels
@timdiels
Apr 13 2017 18:05
@pditommaso Thanks
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:05
workflow.saveMetadata( path )
@timdiels welcome
Karin Lagesen
@karinlag
Apr 13 2017 18:05
yes indeed!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:06
if you can help me to not forget it opening a feature request on GH, I may try to do it
:)
Evan Floden
@evanfloden
Apr 13 2017 18:06
:+1: Nice idea
Mike Smoot
@mes5k
Apr 13 2017 18:11
+1 here too
Alessia
@alesssia
Apr 13 2017 18:50
Hello, this is surely a stupid question :) I am trying to run a nextflow pipeline using a Docker container and the option -with-docker mydocker, but how do I mount the disk? What I would usually do with docker run --volume ~/myfolder:myfolder?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:51
ciao @alesssia not a stupid question at all
NF mounts automatically the host paths required to access all inputs files
so you don't have to!
Alessia
@alesssia
Apr 13 2017 18:53
mmm, so the error I receive saying that the file does not exist does not depend on this...
Paolo Di Tommaso
@pditommaso
Apr 13 2017 18:54
I have no idea w/o showing the error message
Alessia
@alesssia
Apr 13 2017 18:58
I am running a process that uses a bash script. First line creates a log file, as log=\"$workingpath/.log.c2\" then redirect a message on it (echo \"something\" > \$log and what I get is .command.sh: line 8: ./.log.c2 : No such file or directory (but if I run the .command.sh script it works...)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:01
$workingpath is supposed to be the local path ie. . ?
Alessia
@alesssia
Apr 13 2017 19:01
I tried with both local and absolute path
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:02
the NF way is to work in the tasks local dir
so you could just do
log=".log.c2"
echo "something" > \$log
it should work in this way
Alessia
@alesssia
Apr 13 2017 19:04
That does not work on our HPC (but notably it works with the absolute path), trying to have NF working an all system (that is: testing it on my machine now)
the NF script work fine on HPC (without docker) no way of having it running on my laptop (MacBook Pro)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:05
the NF script work fine on HPC (without docker) no way of having it running on my laptop (MacBook Pro)
because I suspect you have used absolute path somewhere
Alessia
@alesssia
Apr 13 2017 19:06
Nein :)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:06
but notably it works with the absolute path
?
Alessia
@alesssia
Apr 13 2017 19:06
it is a command line parameter
that is then added to all the filepath (--outdir=/something/somewhere/)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:07
um
Alessia
@alesssia
Apr 13 2017 19:07
workingpath = params.outdir + "/" + params.prefix
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:07
ah!
any path/file that you use in your process script it must be properly declared as an input
otherwise it may not work with some executor or when using containers
Alessia
@alesssia
Apr 13 2017 19:10
so doing nextflow run pipeline.nf --outdir /something/somewhere/ does not mean declaring it?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:11
nope, that's just passing a parameter
I mean something like
Alessia
@alesssia
Apr 13 2017 19:11
workingpath = params.outdir + "/" + params.prefix
workingdir = file(workingpath)
if( !workingdir.exists() ) {
    if( !workingdir.mkdirs() )     {
        exit 1, "Cannot create working directory: $workingpath"
    } 
}
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:11
process foo {
  input: 
  file x from bar_ch

  """
  cat $x
  """
}
Alessia
@alesssia
Apr 13 2017 19:12
I can't use workingdir that is declared as file(workingpath)?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:12
I mean process input, as showed above
you should never use an absolute path in a process script
Alessia
@alesssia
Apr 13 2017 19:13
mmm, so I need to create a channel for each process just to set up the working dir (I can' reuse the same channel, can I?)
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:13
nope
you don't need to setup the work dir, this is exactly the thing that makes NF different from the other tools
your tasks need to read/write in (their unique) work dir that is assigned by NF
Alessia
@alesssia
Apr 13 2017 19:15
no, wait a second, not sure I have understood. I have 11 process, each of them creates multiple files. Shoud I set up a channel for each of these outputs?
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:16
each for each output yes, but an output can capture one or more file or even a directory
in practice usually you have an output for each process, even if the task produce multiple files
Alessia
@alesssia
Apr 13 2017 19:18
Ok, I need to think about this. Just wondering why it works fine on HPC :confused:
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:20
because, there you force to use the share dir, but I guess if you try to resume the execution with the -resume option won't work
moreover when you use docker
NF needs to mount the file system using the input/output information
thus it won't work anymore
Alessia
@alesssia
Apr 13 2017 19:22
I see, ok. Need to do a few changes then!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:22
to resume, if you want to make sure to have your pipeline portable taking advantage of all NF features
you need to make sure that each task works in its own
seems difficult, but you will realise that simplifies a lot of things
at least I hope so :)
here there's a short tutorial to NF + Docker, you may want to give a look
Alessia
@alesssia
Apr 13 2017 19:24
I looked at it :)
Evan Floden
@evanfloden
Apr 13 2017 19:24
Here is my basic take on it, not sure it helps but:
  • Think of each time a process is executed as a task.
  • Each task lives in it own working directory (which is the the workfolder).
  • With NF, you simply specify which files should go into that that work dir (inputs)
  • and which files you want to use later on (outputs).
  • The channels simply move the inputs/outputs around.
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:24
oops! :D
Alessia
@alesssia
Apr 13 2017 19:24
I think I need a few changes (and I knew that it was a stupid question ;) )
Evan Floden
@evanfloden
Apr 13 2017 19:25
If you share your code on pastebin I’m sure you will find someone to help you out
Alessia
@alesssia
Apr 13 2017 19:25
I think that now that I got where is the issue I can try by myself!
Evan Floden
@evanfloden
Apr 13 2017 19:26
:+1:
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:26
That's a good approach! ;)
Alessia
@alesssia
Apr 13 2017 19:26
will keep it as last shore (in few days)!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:26
ahah
Alessia
@alesssia
Apr 13 2017 19:28
Thank you!
Paolo Di Tommaso
@pditommaso
Apr 13 2017 19:29
Have fun
Alessia
@alesssia
Apr 13 2017 19:32
:D
amacbride
@amacbride
Apr 13 2017 21:01

@pditommaso Is it possible to access the SLURM environment variables from within a template sh script? In particular, I'd like to see the value of SLURM_JOB_NODELIST.

I tried the following, but the environment variable is blank, so I'm assuming I escaped it incorrectly?

#!/bin/bash

echo ""
echo "----"
echo "${task.process}"
echo "execution host: `hostname -s`"
echo "slurm host: \$SLURM_JOB_NODELIST"
It's in the .command.log file, so I figured it must be possible to access it from the template script somehow.
SLURMD_NODENAME=marin
SLURM_GTIDS=0
SLURM_JOB_CPUS_PER_NODE=1
SLURM_JOB_ID=49559
SLURM_JOBID=49559
SLURM_JOB_NAME=nf-envtest
SLURM_JOB_NODELIST=marin
SLURM_JOB_NUM_NODES=1
SLURM_LOCALID=0
SLURM_NNODES=1
SLURM_NODE_ALIASES=(null)
SLURM_NODEID=0
SLURM_NODELIST=marin
Paolo Di Tommaso
@pditommaso
Apr 13 2017 21:32
yes, like any other variable
I don't see why it should not
amacbride
@amacbride
Apr 13 2017 23:59
Here's the output of that first script (this was run with 0.24.1):
----
envtest
execution host: marin
slurm host: 
+ sleep 30