These are chat archives for nextflow-io/nextflow

16th
Jun 2016
Phil Ewels
@ewels
Jun 16 2016 08:34
Hi @pditommaso - basic "what's the best way" question for you..
Paolo Di Tommaso
@pditommaso
Jun 16 2016 08:35
I'm ready :)
Phil Ewels
@ewels
Jun 16 2016 08:35

We recently changed our pipeline to have the following syntax:

output:
file '*.pdf'

instead of

output:
file '*.pdf' into results

(where results was not used by any downstream processes)

Current pipeline I'm looking at has two processes with an output of file '*.pdf'
and I get the following error:
ERROR ~ Channel `*.pdf` has been used twice as an output by process `ngsplot` and process `phantompeakqualtools`
Paolo Di Tommaso
@pditommaso
Jun 16 2016 08:37
I see, this should not happen. Could you please open an issue for this
as workaround just add and into statement to a dummy channel with different names
Phil Ewels
@ewels
Jun 16 2016 08:39
ok, that's what I was about to do but figured that I had just changed syntax to avoid doing that ;)
Paolo Di Tommaso
@pditommaso
Jun 16 2016 08:41
That validation checker has been introduced recently it still needs some refinements. Thanks for the feedback!
Phil Ewels
@ewels
Jun 16 2016 08:41
Yup, no problem! Reported in #182
Paolo Di Tommaso
@pditommaso
Jun 16 2016 08:42
:+1:
Johan Viklund
@viklund
Jun 16 2016 09:06
is it possible to set the NXF_ env vars in the nextflow.config somehow (especially debug and workdir).
Phil Ewels
@ewels
Jun 16 2016 09:09
Should be possible I think? See the env scope docs and environment variables docs
Johan Viklund
@viklund
Jun 16 2016 09:11
I wonder if that will work for the NXF stuff, but I can try
Paolo Di Tommaso
@pditommaso
Jun 16 2016 09:13
In general no because that variable are accessed by the JVM instance
Johan Viklund
@viklund
Jun 16 2016 09:14
Would be nice to be able to specify workdir in the config, I think.
so the config file is read very late in the startup?
Paolo Di Tommaso
@pditommaso
Jun 16 2016 09:15
though NXF_DEBUG should work (but I'm not sure)
instead workDir can be defined in the nextflow.config file
Johan Viklund
@viklund
Jun 16 2016 09:15
ah
thanks
Paolo Di Tommaso
@pditommaso
Jun 16 2016 09:16
it happens here
Johan Viklund
@viklund
Jun 16 2016 09:17
:D
Paolo Di Tommaso
@pditommaso
Jun 16 2016 09:17
;)
Johan Viklund
@viklund
Jun 16 2016 09:18
does that mean it is possible to do nextflow run --workDir someplace ... as well?
Paolo Di Tommaso
@pditommaso
Jun 16 2016 09:19
of course, but it's -w or -work-dir
Johan Viklund
@viklund
Jun 16 2016 09:19
ok
Paolo Di Tommaso
@pditommaso
Jun 16 2016 09:19
have a look to nextflow run -h
Phil Ewels
@ewels
Jun 16 2016 09:23
Side note whilst we're on the topic - is it possible to specify the location of .nextflow.log and .nexflow.history in the same way? We've been discussing whether it's possible to kick off our pipelines without changing working directories..
Paolo Di Tommaso
@pditommaso
Jun 16 2016 09:25
the log files yes, by using -log cli option
for the history file no, it's by definition in the launch directory
We've been discussing whether it's possible to kick off our pipelines without changing working directories..
What do u mean?
Phil Ewels
@ewels
Jun 16 2016 09:28
(Sorry, on my phone so will re-read -h when I'm back at my desk)
We have an automated system that kicks off pipelines when data comes in. One pipeline instance per project.
At the moment we will create a directory and cd to it before kicking off nextflow
But our other pipelines don't do this - just specify full paths for everything to go to and off it goes
Does it make more sense to hold the history file in the work directory?
Paolo Di Tommaso
@pditommaso
Jun 16 2016 09:31
um, the idea is to have a folder for each project in which may need run your pipeline more than one time
so the history track the command line options you have used and above all the run IDs that you need if you want to resume the execution
however there's nothing wrong to not change dir and specify the work-dir and the log file paths as cli options
you will just have a common history will all the runs
also we tend to use a common work-dir that is automatically cleaned after some days, to avoid to accumulate trash
Phil Ewels
@ewels
Jun 16 2016 09:41
ok, yes I think this confirms the argument that I was making - it's better to create a new wd for each project and move there to run nextflow. Then clean up once the project is done / delivered. Then we can go back and resume on failure etc as you say.
Paolo Di Tommaso
@pditommaso
Jun 16 2016 09:43
exactly
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 10:43

Hello!

Error executing process > 'ss_mapping (1)'

Caused by:
  No such property: name for class: _nf_script_ef4314ef

Any idea what I should look at?
Also the work folder contains

work/
└── tmp
    ├── da
    │   └── 8cc869f892fec4c397747359ef3015
    └── e3
        └── 3a2f999d2d36b88e727420450f7e12

this tmp is suspicious!

Paolo Di Tommaso
@pditommaso
Jun 16 2016 10:45
um, may be nextflow-io/nextflow#141 problem?
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 10:48
mmh I've a
set file('*.refined.bam'), file('*.refined.bai') into bam_refinement_output
maybe the wildcard and set do not work together? (Here each * should expand to a single file)
Paolo Di Tommaso
@pditommaso
Jun 16 2016 10:49
I don't think so
could you provide the stack trace
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 10:52
Paolo Di Tommaso
@pditommaso
Jun 16 2016 10:55
can you share also the code or at least the process producing that prob?
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 10:55
actually it also says: -- Check script 'exome.nf' at line: 74
Paolo Di Tommaso
@pditommaso
Jun 16 2016 10:56
noticed that
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 10:56
aroudn those lines I have defined a function
def HEADER = { header, pct ->
"""
echo ""
echo "#######################################"
echo " Starting ${header}"
echo \$(date)
echo "#######################################"
echo ""
echo "${pct}\t\$(date)\t${name}\t${header}" >> ${progress_log_path}
echo "${pct}\t${header}" > ${progress_path}
"""
}
which is the same than defined in another pipeline (which works well)
Paolo Di Tommaso
@pditommaso
Jun 16 2016 10:56
um
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 10:57
line 74 is the """
Paolo Di Tommaso
@pditommaso
Jun 16 2016 10:58
and what is supposed to be name ?
I mean a nxf variable? I don't see it defined
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 10:59
you nailed it
why can't it see that ? :)
Paolo Di Tommaso
@pditommaso
Jun 16 2016 11:00
:)
pair programming works ..
Evan Floden
@evanfloden
Jun 16 2016 11:01
@pditommaso has a 6th sense for bugs
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 11:01
yes I see that! Thanks Paolo
Paolo Di Tommaso
@pditommaso
Jun 16 2016 11:04
Well, I developed that when debugging nondeterministic issues due to concurrency
this was easy ;)
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 11:07
Heisenbugs!
Paolo Di Tommaso
@pditommaso
Jun 16 2016 11:09
:)
I need to remember this
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 11:15

would that work:

errorStrategy task.attempt==1: 'retry' ? 'ignore'

?

Paolo Di Tommaso
@pditommaso
Jun 16 2016 11:17
when it's a dynamic value you need to use a closure or a interpolated string
i.e.
errorStrategy { task.attempt==1 ? 'retry' : 'ignore' }
Phil Ewels
@ewels
Jun 16 2016 11:18
But for that example is it not better to have this?
errorStrategy: 'retry'
maxRetries: 1
maxErrors: '-1'
ah no, as you wouldn't get the ignore
ignore me ;)
Paolo Di Tommaso
@pditommaso
Jun 16 2016 11:19
also note the condition expression is bool expr ? statement1 : statement2
@ewels :)
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 11:20
huh yes ?: typo, thanks for the closure also
It doesn't say Re-submitted process > though
with the
errorStrategy { task.attempt==1 ? 'retry' : 'ignore' }
the rerun process should fail on the first run, be re-submitted, then be ignored (at least that's what I'd like)
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 11:29
instead it is "ignored" on the first run
Paolo Di Tommaso
@pditommaso
Jun 16 2016 12:16
@huguesfontenelle something strange is happening here, I'm checking
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 12:17
nice.
Paolo Di Tommaso
@pditommaso
Jun 16 2016 12:22
@huguesfontenelle coud you please open an issue for that :(
Hugues Fontenelle
@huguesfontenelle
Jun 16 2016 12:52
sure