These are chat archives for nextflow-io/nextflow

9th
Mar 2018
Simone Baffelli
@baffelli
Mar 09 2018 08:07

Good morning. I need help with a strange error that I get when I mix configuration files. I have a "global" nextflow.config file, shared between different pipelines, where i define azlks = 2. Additionally I have a polcal.config that defines some specialized parameters used for certain workflows only, where azlks = 5. When I run a workflow with -c polcal.config I get:

  No signature of method: nextflow.config.ComposedConfigSlurper$_parse_closure5.azlks() is applicable for argument types: (java.lang.Integer) values: [2]

If I remove azlks from polcal.config the error disappears. Is this an expected behavior?

Nvm, there was a typo in the config file
Shellfishgene
@Shellfishgene
Mar 09 2018 10:05
What could the reason be that nf reruns jobs every time I start the pipeline, even with only little changes in later processes? I have everything set to "symlink" in the publishDir directive.
Simone Baffelli
@baffelli
Mar 09 2018 10:12
Very hard to say, but I had a similar problem due to circular dependencies
i.e I was using the same file as input and output
so everytime the pipeline was finished, the file was updated and in the next run nextflow would think the input had changed
Otherwise, be very careful when using the map operator
Shellfishgene
@Shellfishgene
Mar 09 2018 10:18
I think in my case it was because I changed the publishDir to symlink only for this run, so the last was still on "move" and it reran the process...
Bioninbo
@Bioninbo
Mar 09 2018 11:25
Hello. Is there a concise way to assign the same container to multiple processes? Ie something shorter than this:
process {
    $task1.container = params.container1
    $task2.container = params.container1
    $task3.container = params.container1
}
Paolo Di Tommaso
@pditommaso
Mar 09 2018 11:26
not at this time, but it's the goal of this nextflow-io/nextflow#623
Bioninbo
@Bioninbo
Mar 09 2018 11:27
I see thanks
On a different matter, I see many times the cache breaking with different processes and in a seemingly semi-random fashion (some processes break occasionally more than others). Any general suggestion to avoid this?
Paolo Di Tommaso
@pditommaso
Mar 09 2018 11:30
update to the latest version
if not solve there could be a problem in your workflow that should be investigated
Bioninbo
@Bioninbo
Mar 09 2018 11:31
I almost at last version (0.27.6.4775)
Paolo Di Tommaso
@pditommaso
Mar 09 2018 11:31
so second option
Bioninbo
@Bioninbo
Mar 09 2018 11:31
Ok thanks
Paolo Di Tommaso
@pditommaso
Mar 09 2018 11:32
use -dump-hashes what's cause the hash to change
Bioninbo
@Bioninbo
Mar 09 2018 11:35
I see thanks!
It gives a lot of text. Is there an option to just indicate the file(s) that changed which caused the cache to break for a process?
Paolo Di Tommaso
@pditommaso
Mar 09 2018 11:39
that's a low level debugging feature
unfortunately not so user friendly
you need to identify the first hash changing
Bioninbo
@Bioninbo
Mar 09 2018 11:40
Ok sure. It's nice to have this tool already, thanks for the tip
Shellfishgene
@Shellfishgene
Mar 09 2018 13:40
Does Channel.fromPath also return a value that matches the wildcard, like fromFilePairs?
Ah, the FAQ section has this question.
Shellfishgene
@Shellfishgene
Mar 09 2018 13:49
So, I can add .map { file -> tuple(file.baseName, file) } when I create the channel from a path. How would I do it if baseName is not enough and I'd want something like a regex? For example if my file is called sample1_foo_bar.txt and I just need the sample1 part.
Paolo Di Tommaso
@pditommaso
Mar 09 2018 14:42
you need to manipulate that file with an in-line snippet or an helper function