These are chat archives for nextflow-io/nextflow

24th
Mar 2017
Phil Ewels
@ewels
Mar 24 2017 12:19 UTC
Hi @pditommaso - quick (hopefully basic) question about nextflow config loading. Specifically, the order of loading and merging.
The way I read the docs is that the configs are loaded in this order:
  1. Work directory
  2. Script directory
  3. Home directory
  4. -c command line
Is that correct? So any config options I set in ~/.nextflow/config should take precedence over those set in the pipeline script config files?
I haven't done much explicit testing, but I think that we're finding that the home directory config vars are being overwritten by those set in the pipeline script
The pipeline script config is loaded with -profile, if that makes any difference
Paolo Di Tommaso
@pditommaso
Mar 24 2017 12:26 UTC
um, from higher to lower priority
  1. command line
  2. launch dir
  3. script dir
  4. Home dir
This message was deleted
So any config options I set in ~/.nextflow/config should take precedence over those set in the pipeline script config files?
nope, he other way around
that are considered global settings, that you can override with the project config file
Phil Ewels
@ewels
Mar 24 2017 12:28 UTC
fair enough..
my use case is that the pipeline has sensible defaults, which I want to override
and it's annoying to have to use -c all of the time
Is there any way to achieve this? Other than setting a bash alias? :)
Paolo Di Tommaso
@pditommaso
Mar 24 2017 12:30 UTC
um
passwords ?
Maxime Garcia
@MaxUlysse
Mar 24 2017 12:30 UTC
can we load the $NXF_HOME/config in the profile another time ?
Paolo Di Tommaso
@pditommaso
Mar 24 2017 12:31 UTC
what about env variables for that ?
Phil Ewels
@ewels
Mar 24 2017 12:31 UTC
Just now, I have a module load statement in the pipeline config which I want to remove, so that I can run my own version of the software
We also have a base config designed for anyone running on "other systems" that uses local executor, would be good to override that if people want to
plus a few other occasions where I've come across similar things
@MaxUlysse or have two home configs? One for each end of the config loading?
Could get confusing I guess
(the above comment is incorrect, it turns out ;) )
Paolo Di Tommaso
@pditommaso
Mar 24 2017 12:33 UTC
:)
so you want to have a default set of module loads
Maxime Garcia
@MaxUlysse
Mar 24 2017 12:35 UTC

I was more thinking doing like:

profile
 devel {
    includeConfig '$NXF_HOME.config'
    includeConfig 'conf/uppmax-devel.config'
}

But yeah, it's confusing

Paolo Di Tommaso
@pditommaso
Mar 24 2017 12:35 UTC
and override them in a specific case, is that right ?
Phil Ewels
@ewels
Mar 24 2017 12:38 UTC
yes, that's right
(or better still, any generic config settings)
Paolo Di Tommaso
@pditommaso
Mar 24 2017 12:41 UTC
ok, if so what's wrong on having the default setting in the $NXF_HOME/config
and override them with a local config file ?
Karin Lagesen
@karinlag
Mar 24 2017 12:44 UTC
good afternoon
Just wondering if somebody can point me to the right place for figuring out a nextflow project directory structure?
Phil Ewels
@ewels
Mar 24 2017 12:44 UTC
@pditommaso: I'd like the default setting to be present in the pipeline itself
Karin Lagesen
@karinlag
Mar 24 2017 12:44 UTC
I have python scripts, nextflow scripts, config files etc
Phil Ewels
@ewels
Mar 24 2017 12:44 UTC
If these are settings specific to the pipeline
Hi @karinlag - there is a list of Nextflow pipelines here: https://github.com/nextflow-io/awesome-nextflow
We mostly used those for reference when we were just starting out :)
Typically, scripts go into a bin directory in the script directory, which is available to processes (docs)
We then put config files into a config directory, and reference them using -profile [name], with import statements in nextflow.config in the script directory. For example, here
Not sure how others handle configs - this is what we were just discussing above actually
Karin Lagesen
@karinlag
Mar 24 2017 12:49 UTC
:)
Phil Ewels
@ewels
Mar 24 2017 12:49 UTC
As for nextflow scripts - usually there is just a single nextflow script per project. If you call it main.nf then nextflow will be able to automatically pull your pipeline from GitHub (docs)
@pditommaso: Specifically, the 'sensible defaults' in the pipeline will be good for most users. We don't want to require every new user to have to create a new config file for themselves. Nicer to bundle these with the pipeline itself.
Karin Lagesen
@karinlag
Mar 24 2017 12:50 UTC
mucho useful guys :)
Phil Ewels
@ewels
Mar 24 2017 12:50 UTC
In practice this works brilliantly, apart from whenever I want to override those defaults
Karin Lagesen
@karinlag
Mar 24 2017 12:50 UTC
thanks!
Phil Ewels
@ewels
Mar 24 2017 12:51 UTC
No problem @karinlag! Hope that helps :)
Paolo Di Tommaso
@pditommaso
Mar 24 2017 13:04 UTC
I see, I think what's is suggesting Max makes sense in this case
Maxime Garcia
@MaxUlysse
Mar 24 2017 13:05 UTC
I make sense \o/
Phil Ewels
@ewels
Mar 24 2017 13:05 UTC
Only problem with that is that a pipeline can then effectively change the default Nextflow behaviour
which could be kind of confusing for people using it
Paolo Di Tommaso
@pditommaso
Mar 24 2017 13:07 UTC
I tend to agree
I would try to avoid to do too many magic things and keep a reduce set of default
Phil Ewels
@ewels
Mar 24 2017 13:08 UTC
Yeah, I've just dropped a bunch of settings which were NF defaults anyway
Paolo Di Tommaso
@pditommaso
Mar 24 2017 13:08 UTC
even in the case this would mean to replicate some settings across pipelines
IMO it's better to keep things as self-contained as possible without relying on the external system as much as possible
Phil Ewels
@ewels
Mar 24 2017 13:10 UTC
I guess this comes down to the question of who is using the pipeline. All of this is pretty easy if it's just us pipeline authors who are running it. But I'm pushing it as a solution for anyone on our compute cluster, which covers a lot of people. I like the idea that they can just install NF and then run a single command and everything "just works". So need pipeline defaults for module loading etc for that to happen..
Paolo Di Tommaso
@pditommaso
Mar 24 2017 13:10 UTC
I like it as well!
Phil Ewels
@ewels
Mar 24 2017 13:11 UTC
Hmm..
Paolo Di Tommaso
@pditommaso
Mar 24 2017 13:11 UTC
for this I would suggest to use different profiles, having a specific configuration for each target environment
Phil Ewels
@ewels
Mar 24 2017 13:12 UTC
yeah, I was just settling on the same conclusion
we already have a 'dev' profile, I could just strip specifics out of that
ok cool, well I know how this all works now and am no longer mis-informing people ;)
Thanks for the discussion! :+1:
Paolo Di Tommaso
@pditommaso
Mar 24 2017 13:13 UTC
you can even organise it in separate files and then include in different profiles
have a look here
works well :+1:
Paolo Di Tommaso
@pditommaso
Mar 24 2017 13:14 UTC
cool
Karin Lagesen
@karinlag
Mar 24 2017 13:44 UTC
question @ewels , how do you specify a profile? Do you do that on the cmd line? (sorry if this is a faq)
Phil Ewels
@ewels
Mar 24 2017 13:45 UTC
-profile. Docs here
Though if not specified, Nextflow looks for a profile called standard which is the default
Karin Lagesen
@karinlag
Mar 24 2017 13:49 UTC
ah
thanks!
Paolo Di Tommaso
@pditommaso
Mar 24 2017 16:31 UTC
hey, NF documentation is far from perfect, but it's specified in the last paragraph that the standard profile is used by default ;)
Daniel E Cook
@danielecook
Mar 24 2017 16:48 UTC
Anyone see an error like this: No signature of method: java.util.ArrayList.toAbsolutePath() is applicable for argument types: () values: []; I’m using a group tuple in a process and merging with ${bam.join(“ “)} where I am merging a group of bams into one file.
Paolo Di Tommaso
@pditommaso
Mar 24 2017 16:51 UTC
can you post the stack trace? you can find it in the log file
Daniel E Cook
@danielecook
Mar 24 2017 16:58 UTC

`ERROR ~ Error executing process > 'merge_bam (E119)'

Caused by:
No signature of method: java.util.ArrayList.toAbsolutePath() is applicable for argument types: () values: []

-- Check '.nextflow.log' file for details
WARN: Killing pending tasks (10)`

Paolo Di Tommaso
@pditommaso
Mar 24 2017 16:58 UTC
(triple ` and new line) :)
Daniel E Cook
@danielecook
Mar 24 2017 16:58 UTC
ah was trying to do that
Paolo Di Tommaso
@pditommaso
Mar 24 2017 16:58 UTC
however the full stack trace is in the .nextflow.log file
Daniel E Cook
@danielecook
Mar 24 2017 16:58 UTC
yeah it’s kinda big
it’s 1.8 mb
i’ll send a link to it
Paolo Di Tommaso
@pditommaso
Mar 24 2017 16:59 UTC
ok
Oh I see where the trace is now. My bad
Paolo Di Tommaso
@pditommaso
Mar 24 2017 17:00 UTC
no pb
here it is

groovy.lang.MissingMethodException: No signature of method: java.util.ArrayList.toAbsolutePath() is applicable for argument types: () values: []
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:58)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:64)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:54)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:52)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
    at nextflow.extension.FilesEx.complete(FilesEx.groovy:1385)
    at nextflow.processor.TaskConfig.getStoreDir(TaskConfig.groovy:283)
    at sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaBeanProperty.getProperty(MetaBeanProperty.java:59)
    at nextflow.processor.TaskConfig.getProperty(TaskConfig.groovy:90)
    at org.codehaus.groovy.runtime.callsite.PogoGetPropertySite.getProperty(PogoGetPropertySite.java:50)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
    at nextflow.processor.TaskProcessor.checkStoredOutput(TaskProcessor.groovy:758)
    at nextflow.processor.TaskProcessor$checkStoredOutput$1.callCurrent(Unknown Source)
    at nextflow.processor.TaskProcessor.invokeTask(TaskProcessor.groovy:565)
    at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
I think you used a list of files as value for storeDir directive
Daniel E Cook
@danielecook
Mar 24 2017 17:01 UTC
Yes, and the process I am using is:

process merge_bam {

    echo true

    storeDir bam_dir, pattern: '*.bam*'

    cpus cores

    tag { SM }

    input:
        set SM, bam from sample_aligned_bams_use.groupTuple()

    output:
        set val(SM), file("${SM}.bam"), file("${SM}.bam.bai") into merged_SM
        file("${SM}.duplicates.txt") into duplicates_file

    """
    count=`echo ${bam.join(" ")} | tr ' ' '\\n' | wc -l`

    if [ "\${count}" -eq "1" ]; then
        ln -s ${bam[0]} ${SM}.merged.bam
        ln -s ${bam[0]}.bai ${SM}.merged.bam.bai
    else
        sambamba merge --nthreads=${cores} --show-progress ${SM}.merged.bam ${bam.join(" ")}
        sambamba index --nthreads=${cores} ${SM}.merged.bam
    fi

    picard MarkDuplicates I=${SM}.merged.bam O=${SM}.bam M=${SM}.duplicates.txt VALIDATION_STRINGENCY=SILENT REMOVE_DUPLICATES=false
    sambamba index --nthreads=${cores} ${SM}.bam
    “"”
}
Paolo Di Tommaso
@pditommaso
Mar 24 2017 17:02 UTC
how is defined bam_dir ?
Daniel E Cook
@danielecook
Mar 24 2017 17:02 UTC
bam_dir = "/projects/b1059/data/alignments/RIL”
thats in nextflow.config
and I redefine it at the top of main.nf using bam_dir = config.bam_dir
Paolo Di Tommaso
@pditommaso
Mar 24 2017 17:03 UTC
the error doesn't suggest that
wait
you cannot use , pattern: '*.bam*' in the storeDir
Daniel E Cook
@danielecook
Mar 24 2017 17:03 UTC
ah ok
Paolo Di Tommaso
@pditommaso
Mar 24 2017 17:04 UTC
are you confusing it with publishDir ?
Daniel E Cook
@danielecook
Mar 24 2017 17:04 UTC
yes - I switched it to storeDir
from publishDir
Paolo Di Tommaso
@pditommaso
Mar 24 2017 17:04 UTC
remove pattern if so
Daniel E Cook
@danielecook
Mar 24 2017 17:04 UTC
ok i’ll give it a go
So far so good! Before it errored and it’s running so I believe it’s fixed
Thank you
Paolo Di Tommaso
@pditommaso
Mar 24 2017 17:05 UTC
:v:
Phil Ewels
@ewels
Mar 24 2017 17:39 UTC
Hah, sorry @pditommaso - I know it's mentioned! I meant if -profileisn't specified on the command line ;P
Paolo Di Tommaso
@pditommaso
Mar 24 2017 17:40 UTC
ahah, true!
the documentation for the CLI it's completely missing !