Nextflow community chat moved to Slack! https://www.nextflow.io/blog/2022/nextflow-is-moving-to-slack.html
@giannicorik_twitter it is to use in the
workflow.onComplete method
Oh, I think I misunderstand the
size option...
Any way of splitting the fasta file by bytes, without cutting sequences in half?
@AlaaBadredine_twitter I've made another implementation of the A->(B->)->C pattern: https://github.com/micans/nextflow-idioms/blob/master/ab-abc-tap.nf . I think it is the most readable one -- it uses the
tap operator (which I noticed in @giannicorik_twitter 's contribution above). The core looks like this:ch_dummy.flatMap().map { f -> [f.text.trim(), f] }.view()
.tap { ch_AC }
.until { !params.includeB }
.set { ch_AB }
process processB {
input: set val(sampleid), file(thefile) from ch_AB
output: set val(sampleid), file('out.txt') into ch_BC
script: "(echo 'B process'; cat $thefile; md5sum $thefile) > out.txt"
}
ch_AC.until { params.includeB }.mix(ch_BC).set{ ch_C }
@pditommaso this way there are no extra channel names ...
@giannicorik_twitter Thanks for the examples! That's helpful.
Does anyone have suggestions for an idiom like:
C is a process that collects a set of files and creates some output. A outputs a full set of files to C and once they have all arrived C runs. B also outputs a full set of files to C at another time and C runs completely independently of A's input. C creates output/A/files output/B/files
@madkinsz sounds exactly like this: https://github.com/micans/nextflow-idioms/blob/master/collectFile-tuple.nf (with many thanks to @pditommaso as always).
It's kind of complicated, you probably want to combine multiple channels into a single one using some of the nextflow operators ( see here: https://www.nextflow.io/docs/latest/operator.html#combining-operators ) but it's not very clear to me how your 3 channels should merge from your description.
After that it will be something like this:
ch_mergedchannel= \\some nextflow-foo starting from ch_infiles.collect() , ch_infilesFromA , ch_infilesFromB
process C {
input:
file (infiles) from ch_mergedchannel
output:
file ("*.files.extensions") into ch_out
publishDir "output/A" pattern "*.a.txt"
publishDir "output/B" pattern "*.b.txt"
}
@madkinsz I think you want to combine A end B channels to create a channel like this
ch_mergedchannel= [ [A , fileA1.txt] , [A , fileA2.txt] , [B , fileB1.txt] , [B , fileB2.txt] ]so you will have an instance of C with fileA1.txt and all the collected files , a second instance of C with fileA2.txt and all the collected files, and so on... right?
process C {
input:
file (infiles) from ch_infiles.collect()
val (source) file (infiles) from ch_mergedchannel
output:
file ("*.txt") into ch_out
publishDir "output/${source}" pattern "*.${source}.txt"
}
I've been looking at the channel operators, but basically I want to reuse process
C multiple times which seems tricky to do.
First, thank you.
Second, I got the same result, but unsure what I got.
What I would like to happen is the process run 4 times.
When I check the work dir, I see only one.
Please explain.
executor > local (4)
[01/994a1d] process > X (1) [100%] 4 of 4 ✔
>>cd work/01/994a1db445b2f324724cbcdf291d18
~/dev/nextflow/sandbox/work/01/994a1db445b2f324724cbcdf291d18
>>ls
total 8
-rw-r--r-- 1 rithy8 staff 4 Jun 4 08:16 hello.txt
@madkinsz Yes, a process waits for all his channels in input to be completely filled before starting.
If you want to run them separately It's quite uggly but I think the easiest solution is to create two C processes.
If you want to run them separately It's quite uggly but I think the easiest solution is to create two C processes.
process C_A {
input:
file (infiles) from ch_infiles.collect()
file (infiles) from ch_infilesFromA
output:
file ("*.txt") into ch_outCA
publishDir "output/A"
}
process C_B {
input:
file (infiles) from ch2_infiles.collect()
file (infiles) from ch_infilesFromB
output:
file ("*.txt") into ch_outCB
publishDir "output/B"
}
@madkinsz well, you could use a template (see here: https://www.nextflow.io/docs/latest/process.html#template) but personally I don't love it because you don't have all together inside the .nf file
I will probably just let it run at the end when it's all been collected.
@madkinsz you know, I'm starting wandering if I'm wrong ... I'm not so sure a process just wait for all the channels to be completely filled. Which is a good news for you! :P
I mean normally a channel item can be processed without the channel to be completed. I also remember when a process has a
You should try that, I think the process C will start executing the first channel A elements as long as they are ready, one by one.
I mean normally a channel item can be processed without the channel to be completed. I also remember when a process has a
collect channel it , the collected channel needs to be completed before the process can start, but in your case the bottleneck is not in the collected one.You should try that, I think the process C will start executing the first channel A elements as long as they are ready, one by one.