These are chat archives for nextflow-io/nextflow

16th
Nov 2018
Rad Suchecki
@rsuchecki
Nov 16 2018 00:24

You can use the second version of https://www.nextflow.io/docs/latest/operator.html#into which will generate a list of channels. Depending on the application it may be more convenient to use a queue rather than a list.

For example, you generate n channels, and present them as queue from which each downstream process polls:

channelsQueue = inChannel.into(n) as Queue

and in each downstream process:

input:
    ... from (channelsQueue.poll())
Maxime HEBRARD
@mhebrard
Nov 16 2018 09:45
hello, I have a strange issue, I run STAR aligner from a VirtualBox on windows using a nextflow script ... the reads of my samples are processed as expected (or so it seams for now) but the RAM use by the virtual machine grow linearly ... sounds like when a mapping process is finished, the RAM is not cleared and the next process "pile up" ... any good advice on what might goes wrong ?
Paolo Di Tommaso
@pditommaso
Nov 16 2018 10:02
it may be the java garbage collector
try to set min max heap size
Tobias Neumann
@t-neumann
Nov 16 2018 10:50
@pditommaso I was launching a nextflow pipeline on AWS from my local machine and for 50 samples the memory consumption apparently scaled linearly up to 50GB. did I misconfigure something? because this usage just for a submission jobs is totally off
Maxime Garcia
@MaxUlysse
Nov 16 2018 10:54
@t-neumann It's easy to make a github page using jekyll or hugo
There are some themes that are already ready for academic, such as https://academicpages.github.io/
Evan Floden
@evanfloden
Nov 16 2018 10:57
@rsuchecki The channel queue idea is super nice idea. Did a quick test and seems to work well.
Tobias Neumann
@t-neumann
Nov 16 2018 10:58
@MaxUlysse Yes I already looked at your github page, it looks beautiful. I think I would setup something like this
micans
@micans
Nov 16 2018 11:13
@evanfloden I'm trying to find documentation for poll() ... is it a Groovy thing?
Rad Suchecki
@rsuchecki
Nov 16 2018 12:07
Java @micans so groovy too
Rad Suchecki
@rsuchecki
Nov 16 2018 12:22
@evanfloden :smile: Used at some point to run alignments on the same input with multiple tools. Worked really well, perhaps readability suffered a bit. Now prefer to use single, generic align process with multiple templates, but that is specific to my use case.
Paolo Di Tommaso
@pditommaso
Nov 16 2018 12:41
it's a brilliant idea, actually it could be possible to refine it a bit and remove the need of the as Queue and explicit .poll() invocation
Paolo Di Tommaso
@pditommaso
Nov 16 2018 13:04
what something like this?
Channel 
    .from(1,2,3,4)
    .fork(2)
    .set{ ch }

process foo {
    input: val x from ch
    exec: 
    println "$task.process $x"
}

process bar {
    input: val x from ch
    exec: 
    println "$task.process $x"
}
Rad Suchecki
@rsuchecki
Nov 16 2018 13:47
That'd be really neat.
Rahul Pisupati
@rbpisupati
Nov 16 2018 14:35
hello, i have a question. i have a channel with all the file names. How do I pass all the files to to a command as "-variant file_1 -variant file_2 -variant file_3"?
I did try using collect in the operator! and thinking of trying to use buffer but how do I add "-variant" in the string.
I want to run this https://software.broadinstitute.org/gatk/documentation/tooldocs/3.8-0/org_broadinstitute_gatk_tools_walkers_variantutils_GenotypeGVCFs.php
Paolo Di Tommaso
@pditommaso
Nov 16 2018 14:47
have a look to an example here
Nurlan Kerimov
@kerimoff
Nov 16 2018 19:18

Hello dear community, Some help needed here

I did not want add this issue to git since it can be my lack of experience with containers.
I am trying to run NF-core\rnaseq in our local HPC (slurm)

I use local singularity image which I pulled and stored in HPC location which all nodes can access. (I shelled to container and checked, all software needed is there)
I get weird error like it tries to change directory in the container, LOG FILE

Caused by:
  Process `trim_galore (SAMEA3234534)` terminated with an error exit status (127)

Command executed:

  trim_galore --paired --fastqc --gzip     SAMEA3234534_1.fastq.gz SAMEA3234534_2.fastq.gz

Command exit status:
  127

Command output:
  (empty)

Command error:
  /bin/bash: line 0: cd: /gpfs/hpc/home/a72094/projects/nextflow_ws/rnaseq-master/work/cd/2bc80a82fbb51ea5f3421c311854a0: No such file or directory
  /bin/bash: .command.stub: No such file or directory

Here is the repo of all needed files for analysis
which includes: config files which I added and changed, log file, created job files (under work/), the command I run.

Any help is appreciated :)

Tobias "Tobi" Schraink
@tobsecret
Nov 16 2018 21:27
I just get Killed when running my NextFlow executor, does that mean I ran out of memory or is it being killed externally?
Interestingly enough the pending jobs are not killed.
[53/6f6e96] Submitted process > align_reads_bwa (SampleID ERS142857 SampleSubID ERR263689)
Killed
[user@log-0]$