These are chat archives for nextflow-io/nextflow

9th
Jan 2018
Karin Lagesen
@karinlag
Jan 09 2018 08:39
good morning!
I get the following warning that I don't quite understand
WARN: Unknown directive normal for process run_fastqc_eval
just updated nextflow, and it persists
and I can't find anything about a directive normal in the docs?
Paolo Di Tommaso
@pditommaso
Jan 09 2018 09:01
you can't find anything because normal is not a nextflow directive .. ;)
Phil Ewels
@ewels
Jan 09 2018 14:49
Hi @pditommaso! Has v27 just gone live? :D
A travis build just fell over (after it succeeded moments earlier in a PR), and I think it's because of a nextflow change
Paolo Di Tommaso
@pditommaso
Jan 09 2018 14:50
yes, fixing some typos in the docs before announcing it :)
Phil Ewels
@ewels
Jan 09 2018 14:50
has workflow.revision been removed?
(or is there something else going on here?)
Paolo Di Tommaso
@pditommaso
Jan 09 2018 14:51
umm, no
now it checks for strictly for missing configuration attributes
Phil Ewels
@ewels
Jan 09 2018 14:52
This is the build from a few moments earlier that passed with v0.26.4: https://travis-ci.org/SciLifeLab/NGI-ChIPseq/builds/326813731
Paolo Di Tommaso
@pditommaso
Jan 09 2018 14:52
so there was/is an error in your config
Phil Ewels
@ewels
Jan 09 2018 14:53
ah, ok. I guess workflow isn't yet defined when nextflow.config is parsed?
Paolo Di Tommaso
@pditommaso
Jan 09 2018 14:53
nope
Phil Ewels
@ewels
Jan 09 2018 14:53
And I'd have never spotted this as it would return false and default to latest as expected
Paolo Di Tommaso
@pditommaso
Jan 09 2018 14:54
basically it was reporting an empty value ..
Phil Ewels
@ewels
Jan 09 2018 14:54
Is there a way to know what the -r v1.3.1 or whatever command line flag was specified when parsing the config?
Paolo Di Tommaso
@pditommaso
Jan 09 2018 14:54
hence, the new version as already fixed a bug in your pipeline :satisfied:
Phil Ewels
@ewels
Jan 09 2018 14:54
the idea was to automatically pull the container tagged with the same release version
Paolo Di Tommaso
@pditommaso
Jan 09 2018 14:55
Is there a way to know what the -r v1.3.1 or whatever command line flag was specified when parsing the config?
unfortunately no
Phil Ewels
@ewels
Jan 09 2018 14:56
hmm, sad times.
I wonder if I can get the same effect by moving the logic into the pipeline itself somehow
will be messier though :(
Paolo Di Tommaso
@pditommaso
Jan 09 2018 14:58
@emi80 is doing something similar in the grape pipeline, see here
but not sure if I would suggest that, I found it overkill
IMO the configuration should be *human* readable
Phil Ewels
@ewels
Jan 09 2018 15:00
yeah, agreed. Can I just add some squiggly brackets or something to defer execution until later?
I guess I need to do a bit of trial and error :)
Maybe squiggly brackets + defining the container under the process scope may work...
Emilio Palumbo
@emi80
Jan 09 2018 15:01
hey guys! Happy new year!
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:01
happy new year !
Maybe squiggly brackets + defining the container under the process scope may work..
yes, you need a closure resolved by a process
Emilio Palumbo
@emi80
Jan 09 2018 15:02
@pditommaso I see what you mean... I could not find a cleaner solution for this. It's a bit unreadable but it works well for me :)
If you have a better solution I will be extremely grateful
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:03
well done!
Emilio Palumbo
@emi80
Jan 09 2018 15:03
:)
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:03
I haven't :D
Phil Ewels
@ewels
Jan 09 2018 15:04
it's a very efficient solution :)
Emilio Palumbo
@emi80
Jan 09 2018 15:04
...and ugly
:D
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:04
you are too smart for my standard :sweat_smile:
Phil Ewels
@ewels
Jan 09 2018 15:04
slightly different to what I'm trying to achieve - I'm trying to get at a variable that is defined in a different scope
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:05
are not using that to define the container version ?
Phil Ewels
@ewels
Jan 09 2018 15:05
yes
to tie the repository and container version / tags together
so if someone runs v1.2 of the pipeline, it uses the v1.2 container tag
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:05
hence is not supposed by the container in the process ?
Phil Ewels
@ewels
Jan 09 2018 15:06
no it is - I use that variable in other config files, eg: https://github.com/SciLifeLab/NGI-ChIPseq/blob/master/conf/uppmax.config#L18
Defining it as a variable first was helpful as I wanted to use it for both docker and singularity, differently in different profiles
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:07
but you should still be able to reduce it to a closure def such as
container = { "scilifelab/ngi-chipseq:${workflow.revision ? workflow.revision : 'latest'" }
Phil Ewels
@ewels
Jan 09 2018 15:10
Awesome :+1:
Yes that's what I was hoping for above with my "squiggly brackets" comment
ace - will try this out soon
thanks!
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:12
welcome
nice news of the days
Nextflow has been approved as a GitHub supported language :tada: :tada: :tada:
github/linguist#3870
Ashley S Doane
@DoaneAS
Jan 09 2018 15:13

hi, I was hoping someone could help with this operator syntax. I have several processes that each output a set, like:

   output:
    set Sample, file('$Sample.bam')

Then I match the sample name with files from each process.

    finalbamforqc.mix(nsortedbamforqc)
    .mix(broadpeakqc)
    .mix(finalbedqc)
    .mix(sortbamqc)
    .mix(insertionTrackbw)
    .mix(picardcomplexity)
    .mix(pbcqc)
    .mix(dupqc)
    .mix(frips)
    .groupTuple(sort: true)
    .set{ qcinputs }

This emits a set of groupTuples as one output, but I want each groupTuple to be emit seperately.

I don't want to use the set, but not sure what type of operator to use
@pditommaso 👏🏻
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:16
but I want each groupTuple to be emit seperately.
what do you mean exactly? separate channels ?
Ashley S Doane
@DoaneAS
Jan 09 2018 15:18
same channel, once for each groupTuple which are files for the same sample
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:19
groupTuple should do that
Ashley S Doane
@DoaneAS
Jan 09 2018 15:20
ok, just take off the last line, and name the channel, like
    qcfiles = finalbamforqc.mix(nsortedbamforqc)
        .mix(broadpeakqc)
        .mix(finalbedqc)
        .mix(sortbamqc)
        .mix(insertionTrackbw)
        .mix(picardcomplexity)
        .mix(pbcqc)
        .mix(dupqc)
        .mix(frips)
        .groupTuple(sort: true)
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:22
this it's the identical to use .set
Phil Ewels
@ewels
Jan 09 2018 15:23

Nextflow has been approved as a GitHub supported language

Super nice! Thanks for this!

Means we can start removing all of the groovy formatting tags at the head of every file :)
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:23
exactly
Phil Ewels
@ewels
Jan 09 2018 15:24
and Atom editor highlighting too - Christmas coming super early! (or late?) :christmas_tree: :gift:
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:24
still vey basic, but at least recognised the process
Ashley S Doane
@DoaneAS
Jan 09 2018 15:25
for some reason, the process that takes that as input seems to only run once, even if there are 20 samples...
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:26
put a .view() after .groupTuple
Ashley S Doane
@DoaneAS
Jan 09 2018 15:26
ok, yeah I had that before actually
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:26
and is it fine ?
Ashley S Doane
@DoaneAS
Jan 09 2018 15:29
the output is correct- like a list [Sample1, (Sample1.file1, Sample1.file2)]
but for some reason the next process only ran on the first list. So no Sample2. Maybe I need to add a collect statement or something in that next qc process
in case it's helpful, here's the actual script https://github.com/DoaneAS/atacflow/blob/master/main.nf
Paolo Di Tommaso
@pditommaso
Jan 09 2018 15:56
so the problem it must be in the composition of the process inputs
what is the process name ?
Ashley S Doane
@DoaneAS
Jan 09 2018 15:58
thanks so much for taking a look- process is atacqc
end of script
Paolo Di Tommaso
@pditommaso
Jan 09 2018 16:00
I see, I guess all of these are just a single file
replace that block with
dnase = file(params.DNASE)
tssenrich = file(params.TSS_ENRICH)
prom = file(params.PROM)
enh = file(params.ENH)
reg2map = file(params.REG2MAP)
roadmapmeta = file(params.ROADMAP_META)
ref = file(params.REF)
blackqc = file(params.BLACK)
it urges a FAQ entry about this ..
Ashley S Doane
@DoaneAS
Jan 09 2018 16:01
ok. yeah that's the references for qc stuff
those are all from encode, but yes.
ok thanks!!
Ashley S Doane
@DoaneAS
Jan 09 2018 20:38
@pditommaso thanks so much, that was it!
Paolo Di Tommaso
@pditommaso
Jan 09 2018 20:38
:+1: