These are chat archives for nextflow-io/nextflow

8th
Mar 2019
Rad Suchecki
@rsuchecki
Mar 08 01:12
Maxime Vallée
@valleem
Mar 08 07:14
Hello! Is it possible to tailor executor.queueSize per process? I have one long process that I have to submit to a particular Slurm partition, but it requires a lower job count per user than the regular Slurm partition. And all theother processes are launched on the regular Slurm partition, which able users to submit much more jobs.
micans
@micans
Mar 08 10:16
@rsuchecki we are new to this, and nothing seems to work; also we run singularity 2.5. Oh my, I just checked the version, and on two different machines I get 2.5.2 and then 3.1.0. Gaaaahhh. The observation of -B $PWD -B /same/path/as/pwd/but/spelled/out in .command.run seems a clear starting point. Time for me to talk to admins maybe
micans
@micans
Mar 08 11:17
Thanks, (the 3.1.0 was a development machine that was acccidentally allocated to me)
Rad Suchecki
@rsuchecki
Mar 08 11:18
pipelines used to work across a number of singularity versions 2.5 - 3.0 and automounts true was not causing any problems, for now I use a workaround
automounts true with scratch true
as in this case $PWD is not the workDir
micans
@micans
Mar 08 11:20
aha. difficult to use for me, our setup is centered on /lustre, scratch true will require a lot more instrumentation of resources
Rad Suchecki
@rsuchecki
Mar 08 11:20
well, you can point scratch to be anywhere
micans
@micans
Mar 08 11:21
aha. good point.
have to stop ahaing now
thanks ... will try that!
Rad Suchecki
@rsuchecki
Mar 08 11:21
:grinning:
micans
@micans
Mar 08 11:22
when did things change for you? I wonder if it can be NF change that triggered it?
Rad Suchecki
@rsuchecki
Mar 08 11:24
could be, it crept-up on me I should probably do a few runs of one of the pipelines, checking out a tag that did work
Paolo Di Tommaso
@pditommaso
Mar 08 11:38
no, NF never changed how container mounts are managed (recently)
micans
@micans
Mar 08 11:40
I tried @rsuchecki 's suggestion; now I get an error that the input files do not exist; which is a mount issue. I have a process that does ln -s of files in a user-specified directory as the very first step. Yesterday Vlad had a related error with fromPath. In my case, I can see that the mount may be an issue.
I guess if I put those files in the same places as the scratch value it might work ... is there another way around this?
Rad Suchecki
@rsuchecki
Mar 08 11:42
thanks @pditommaso
you can explicitly bind a location @micans
micans
@micans
Mar 08 11:42
sounds good ... how? :-)
Rad Suchecki
@rsuchecki
Mar 08 11:43
singularity.runOptions = '-B /path'
micans
@micans
Mar 08 11:44
@pditommaso a little above I described -B $PWD -B /same/path/spelled/out ... could that be an issue, or is it a red herring?
@rsuchecki thanks a lot. I should have been able to find that, but well, thanks!
Rad Suchecki
@rsuchecki
Mar 08 11:44
:+1:
Rad Suchecki
@rsuchecki
Mar 08 11:50
argghh can't even test earlier versions of singularity on our cluster as module load of any version earlier than 2.6 loads 2.6 (I remember hearing something about security issues in the earlier versions)
micans
@micans
Mar 08 11:55
(about comment before red herring; yes, because it only gives a warning, not an error).
Rad Suchecki
@rsuchecki
Mar 08 12:00
hmm, looks like a FATAL error to me - the misspelled error message as discussed in https://github.com/sylabs/singularity/issues/1469#issuecomment-469129088 @micans
micans
@micans
Mar 08 12:06
going for lunch now, will check this; we had an instance where it very much looked like a warning, but need to doublecheck. With your suggestions my pilot pipeline is now running ... yay :smile_cat:
Rad Suchecki
@rsuchecki
Mar 08 12:13
:v:
micans
@micans
Mar 08 13:57
.... apart from a recolation error for salmon. and a missing libz.so.1 for featureCounts. I guess it's why I have a job :eyes:
Matthew Pocock
@drdozer
Mar 08 14:58
hi - is there a nextflow conference? or an anual conference that tends to have a nextflow/workflows paper track?
Yasset Perez-Riverol
@ypriverol
Mar 08 15:00
Hi all, we have created a list of nextflow workflows https://github.com/BioContainers/workflows/blob/master/workflows.csv from gituhb. Please, if you don’t find your project you can do a PR and we will include them.
KochTobi
@KochTobi
Mar 08 15:13
@ypriverol Awesome
Yasset Perez-Riverol
@ypriverol
Mar 08 15:14
Is a really simple mechanism to keep all nf workflows for new people, for training or other purpouses like indexing
Qi ZHAO
@likelet
Mar 08 15:31
@ypriverol great
Paolo Di Tommaso
@pditommaso
Mar 08 17:36
@drdozer we organise an annual event in barcelona, this year is going to be 19-20 Sept
Olga Botvinnik
@olgabot
Mar 08 20:15

How can one combine multiple glob commands and multiple s3 inputs? I'd like to run fromFilePairs on multiple s3 files but the R1 and R2 files may be specified differently, e.g. {R1,R2}*.fastq.gz or _{1,2}.fastq.gz.

Here's what I have so far:

def paths = file('directories.txt').readLines().findAll { it.size()>0 }

println paths


// --- Try both glob patterns ---
// This is the Illumina style R1/R2 specification
Channel
    .from(paths)
    .map{ it + "**_{R1,R2}_*.fastq.gz"}
    .set{ illumina_style }

// This is the SRA style R1/R2 specifiation
Channel
    .from(paths)
    .map{ it + "**_{1,2}.fastq.gz"}
    .set{ sra_style }


paths_concatenated = illumina_style.concat(sra_style)


Channel
    .from(paths_concatenated)
    .map{ fromFilePairs(it) }
    .println()
Which gives this error:
 ✘  Fri  8 Mar - 12:09  ~/code/nf-kmer-similarity   origin ☊ master ↑1 401● 
  nextflow run main.nf -resume
N E X T F L O W  ~  version 19.01.0
Launching `main.nf` [maniac_murdock] - revision: 043e7f65f8
[s3://olgabot-maca/sra/homo_sapiens/smartseq2_quartzseq/, s3://olgabot-maca/sra/danio_rerio/smart-seq/whole_kidney_marrow_prjna393431/]
samples_ch
ERROR ~ No signature of method: fromFilePairs() is applicable for argument types: (groovyx.gpars.dataflow.DataflowQueue) values: [DataflowQueue(queue=[DataflowVariable(value=s3://olgabot-maca/sra/homo_sapiens/smartseq2_quartzseq/**_{R1,R2}_*.fastq.gz), DataflowVariable(value=s3://olgabot-maca/sra/danio_rerio/smart-seq/whole_kidney_marrow_prjna393431/**_{R1,R2}_*.fastq.gz), DataflowVariable(value=s3://olgabot-maca/sra/homo_sapiens/smartseq2_quartzseq/**_{1,2}.fastq.gz), DataflowVariable(value=s3://olgabot-maca/sra/danio_rerio/smart-seq/whole_kidney_marrow_prjna393431/**_{1,2}.fastq.gz), DataflowVariable(value=groovyx.gpars.dataflow.operator.PoisonPill@25a1f0e2)])]

 -- Check script 'main.nf' at line: 25 or see '.nextflow.log' file for more details
ERROR ~ Cannot get property 'samples_ch' on null object

 -- Check script 'main.nf' at line: 42 or see '.nextflow.log' file for more details
Olga Botvinnik
@olgabot
Mar 08 23:20
Screen Shot 2019-03-08 at 3.19.56 PM.png
Anyone have experience with tee: .command.log: Permission denied and their kin errors for AWS batch?
Is the user that's logged in with Nextflow + AWS batch not the ec2-user? is it root? how can we specify this?