These are chat archives for nextflow-io/nextflow

10th
Sep 2018
tbugfinder
@tbugfinder
Sep 10 2018 06:01
I'm trying to zcat ~100k input filenames driven by an output_channel.collect() method. Unfortunately that's too much input for bash. Any idea how to get this executed?
process finish {
file summarylist from outputchannel.collect()
"""
for i in $summarylist;
        do
                zcat -c \$i >> all.in.one
        done
"""
}
Rad Suchecki
@rsuchecki
Sep 10 2018 07:14
This message was deleted
tbugfinder
@tbugfinder
Sep 10 2018 07:15
As far as I understand $summarylist is already too long.
Paolo Di Tommaso
@pditommaso
Sep 10 2018 07:17
yeah, because all file names will be listed in the task launcher
if that files are in the same dir, use the directory name as input
Paolo Di Tommaso
@pditommaso
Sep 10 2018 08:22
in alternative, you can gather all file names using outputchannel.collectFile { it.toUriString() } then process that file
micans
@micans
Sep 10 2018 13:40
#853 :cry: :laughing:
Paolo Di Tommaso
@pditommaso
Sep 10 2018 13:41
what's up ?
micans
@micans
Sep 10 2018 13:41
It's beenclosed
Paolo Di Tommaso
@pditommaso
Sep 10 2018 13:41
ah-ah ! :joy:
Vladimir Kiselev
@wikiselev
Sep 10 2018 13:42
:cry:
Sorry Paolo...
Paolo Di Tommaso
@pditommaso
Sep 10 2018 13:43
no problem! happy you have solved it
Vladimir Kiselev
@wikiselev
Sep 10 2018 13:43
thanks a lot for you help!
Paolo Di Tommaso
@pditommaso
Sep 10 2018 13:43
you are welcome
tbugfinder
@tbugfinder
Sep 10 2018 17:01
Which is the proper syntax to add newline (newLine: true) to the above outputchannel.CollectFile {..} function? Secondly, as I'm on AWS Batch each filename Uri prefix is s3://. Does it mean that all files should be copied using aws s3 cp or which would be the nextflow approach?
Paolo Di Tommaso
@pditommaso
Sep 10 2018 17:02
Which is the proper syntax to add newline
you said that, newLine: true
Secondly, as I'm on AWS Batch each filename Uri prefix
yes, you will to handle the download on your own
tbugfinder
@tbugfinder
Sep 10 2018 17:09
hm, I get this:
N E X T F L O W  ~  version 0.32.0-SNAPSHOT
Launching `./../main.nf` [kickass_laplace] - revision: e3282e2d48
ERROR ~ expecting '}', found ',' @ line 128, column 73.
   collectFile { it.toUriString(), newLine:
                                 ^

1 error
Paolo Di Tommaso
@pditommaso
Sep 10 2018 17:09
I see, no, it should be
channel. collectFile(newLine: true) { it.toUriString() }
tbugfinder
@tbugfinder
Sep 10 2018 17:38
thank you
Paolo Di Tommaso
@pditommaso
Sep 10 2018 17:39
welcome