These are chat archives for nextflow-io/nextflow

25th
Jul 2016
Mokok
@Mokok
Jul 25 2016 12:02
hello
Mokok
@Mokok
Jul 25 2016 12:12

I quickly read the article you linked me Friday. There were some good information to get ! Thk
And then i saw the NextFlow limitation about in/out size across channels. Can you tell me more about this please ?

Also asked myself (haven't tested yet) how to configure NextFlow to run things across network (as locally but on another machine, distant torque/pbs machine, etc...). Is there a NextFlow agent/daemon ?

Paolo Di Tommaso
@pditommaso
Jul 25 2016 12:25
There's not such in/out size across channels
what are u referring exactly?
when using grid engine such as SGE, SLURM, PBS, etc nextflow doesn't need any daemon
in the case you want to deploy in a cluster without that engines, you can use an embedded nextflow daemon, see here http://www.nextflow.io/blog/2015/mpi-like-execution-with-nextflow.html
Mokok
@Mokok
Jul 25 2016 12:32
i was referring to "cannot handle large stdout | stdin pipes across channels" (i may have misunderstood)
i'm quoting, so you can ctrl+F it
Paolo Di Tommaso
@pditommaso
Jul 25 2016 12:32
ah, ok
that's enough to redirect the stdout to a file
Mokok
@Mokok
Jul 25 2016 12:36
damn i read so fast i haven't paid attention to the "std" part of stdout/stdin", my bad it wasn't about "normal" inputs/outputs, sry
Paolo Di Tommaso
@pditommaso
Jul 25 2016 12:37
yep
Mokok
@Mokok
Jul 25 2016 12:37
that's the evil side of being able to discuss with you so easily i guess
Paolo Di Tommaso
@pditommaso
Jul 25 2016 12:37
it could ;)
Mokok
@Mokok
Jul 25 2016 14:06
is there a way to write the to-be-executed-task's script in groovy ?
Paolo Di Tommaso
@pditommaso
Jul 25 2016 14:06
of course
Mokok
@Mokok
Jul 25 2016 14:10
ok that's what i found...but i get a strange "No such variable " error :/
  1 #!/user/#!/user/bin/env nextflow
  2
  3 textFile = Channel
  4         .fromPath('/home/proactiveuser/nextFlow/data/dataTest1')
  5
  6
  7 process getTheFileAndSplitLines {
  8         input:
  9         each line from textFile.splitText()
 10
 11         output:
 12         val line into lines_channel
 13
 14         exec:
 15         print "line = $line"
 16 }
 17
 18 process extractContent {
 19         input:
 20         var line from lines_channel
 21
 22         output:
 23         var sentence into sentence_channel
 24
 25         exec:
 26         String[] content=line.split("\\s*:{1}\\s")
 27         println content
 28 }
ofc i'm surely doing it the wrong way, could you help me ?
(i'm just splitting a file, containing content such as "Line 1 : foo" and then splitting on " : ")
Paolo Di Tommaso
@pditommaso
Jul 25 2016 14:13
I guess is line 22 sentence is not declared in the exec section
you are supposed to output a variable defined in the exec part
also line 9
you don't need each there
Mokok
@Mokok
Jul 25 2016 14:14
the variable pointed by the error is "line" that's what i find strange
Paolo Di Tommaso
@pditommaso
Jul 25 2016 14:15
use val line from textFile.splitText()
line 1 is messed up
it should be #!/usr/bin/env nextflow
Mokok
@Mokok
Jul 25 2016 14:15
woop ! how did i missed that :o
nvm was due to copy/paste
Mokok
@Mokok
Jul 25 2016 14:21
now telling me line at L9 "no such variable"
each keyword was fixing
Paolo Di Tommaso
@pditommaso
Jul 25 2016 14:25
working for me
paste your code
Mokok
@Mokok
Jul 25 2016 14:27
  1 #!/user/bin/env nextflow
  2
  3 textFile = Channel
  4         .fromPath('/home/proactiveuser/nextFlow/data/dataTest1')
  5
  6
  7 process getTheFileAndSplitLines {
  8         input:
  9         var line from textFile.splitText()
 10
 11         output:
 12         val line into lines_channel
 13
 14         exec:
 15         print "line = $line"
 16 }
Paolo Di Tommaso
@pditommaso
Jul 25 2016 14:28
var is not a keyword
it should be val
Mokok
@Mokok
Jul 25 2016 14:29
damn all this time i was thinking with "var"
just like in C#
Paolo Di Tommaso
@pditommaso
Jul 25 2016 14:29
:)
there are a lot of languages out there ;)
Mokok
@Mokok
Jul 25 2016 14:30
yeah, i'm jumping from one project to an other, it's a bit complicated when there are sneaky variant such as this ^^
Paolo Di Tommaso
@pditommaso
Jul 25 2016 14:31
you need just to get used to the syntax
Mokok
@Mokok
Jul 25 2016 14:33
Yep, I apologize for the inconvenience
Paolo Di Tommaso
@pditommaso
Jul 25 2016 14:33
no pb
Chris Fields
@cjfields
Jul 25 2016 21:40
@pditommaso is there an idiom for working on a set of files from prior analyses? Such as MultiQC? I tried the version from https://github.com/SciLifeLab/NGI-MethylSeq/blob/master/main.nf#L333 but it doesn't work. Work directory simply has a directory with a file containing the string output of toList()
Chris Fields
@cjfields
Jul 25 2016 21:50
I'll see if I can whittle it down to a small test case. But it seems, when using the code in Phil Ewels' NF example there, the toList() gets coerced into a literal string like:
[/home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/94/6e6d0dc4fd2df55fb5ae693d50d0ec/N10_GAGTGGAT_L00M_R1_001_fastqc.html, /home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/94/6e6d0dc4fd2df55fb5ae693d50d0ec/N10_GAGTGGAT_L00M_R1_001_fastqc.zip, /home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/94/6e6d0dc4fd2df55fb5ae693d50d0ec/N10_GAGTGGAT_L00M_R2_001_fastqc.html, /home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/94/6e6d0dc4fd2df55fb5ae693d50d0ec/N10_GAGTGGAT_L00M_R2_001_fastqc.zip]
and that string is the input file. For example, this is my working directory (from one example run):
-system-specific-4.1$ ll work/9b/a643d2fdfed250c9d5440b98ec56a9/2016-07-25-seq-QC-and-trimming/fastqc-raw/
total 128
lrwxrwxrwx 1 cjfields hpcbio_mshook 114 Jul 25 15:01 input.1 -> /home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/tmp/1a/cf38072be407c9d4f407f4a224bd4b/input.1
lrwxrwxrwx 1 cjfields hpcbio_mshook 114 Jul 25 15:01 input.2 -> /home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/tmp/ff/b76cacc408fd0ce270adda54469276/input.2
-system-specific-4.1$ cat work/9b/a643d2fdfed250c9d5440b98ec56a9/2016-07-25-seq-QC-and-trimming/fastqc-raw/input.1
[/home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/94/6e6d0dc4fd2df55fb5ae693d50d0ec/N10_GAGTGGAT_L00M_R1_001_fastqc.html, /home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/94/6e6d0dc4fd2df55fb5ae693d50d0ec/N10_GAGTGGAT_L00M_R1_001_fastqc.zip, /home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/94/6e6d0dc4fd2df55fb5ae693d50d0ec/N10_GAGTGGAT_L00M_R2_001_fastqc.html, /home/groups/hpcbio_shared/mshook/2016-07-19-reanalysis/results/work/94/6e6d0dc4fd2df55fb5ae693d50d0ec/N10_GAGTGGAT_L00M_R2_001_fastqc.zip]
Paolo Di Tommaso
@pditommaso
Jul 25 2016 22:12
This message was deleted
@cjfields can u paste your code snippet ?
Paolo Di Tommaso
@pditommaso
Jul 25 2016 22:19
anyway it looks you didn't declare the input as a file type
for example file ('fastqc/*') from fastqc_results.toList() at this line
Paolo Di Tommaso
@pditommaso
Jul 25 2016 22:27
also, and more likely I think the problem is that that channel has not been created by using the fromPath method
for example:
my_data_files = Channel.fromPath('/your/path/N10_GAGTGGAT_L00M_R1_001_fastqc.*')

process foo {
  input: 
  file x from my_data_files.toList()

  """
  echo $x
  """
}