nextflow-io/nextflow

Nextflow community chat moved to Slack! https://www.nextflow.io/blog/2022/nextflow-is-moving-to-slack.html

Tobias "Tobi" Schraink

@tobsecret

@sstadick I typically write bash scripts that run nextflow with params specified

Paolo Di Tommaso

@pditommaso

Just specify a yaml or json file

Seth

@sstadick

:thumbsup: I didn't realize it was going to be that simple.

Paolo Di Tommaso

@pditommaso

:stuck_out_tongue_closed_eyes:

Shellfishgene

@Shellfishgene

@micans Thanks for the update!

Chadi Saad

@chadisaad

Good morning ! How can I get easly for a specific nextflow job, the list of all child process (executed via slurm) ?

Pierre Lindenbaum

@lindenb

following this conversation on twitter with @pditommaso ( https://twitter.com/yokofakun/status/1159072968158457856 ) I'm trying to implement some new operators. It's not the final design of the source, (I don't think using the notation 'splitter' is a good choice because we don't really read any line). However , it doesn't work.
I implemented a splitter: https://github.com/lindenb/nextflow/blob/pl_htsjdk_op/modules/nextflow/src/main/groovy/nextflow/extension/htsjdk/SamplesSplitter.groovy
I use this splitter in OperatorExt : https://github.com/lindenb/nextflow/blob/pl_htsjdk_op/modules/nextflow/src/main/groovy/nextflow/extension/OperatorEx.groovy#L1623
I compile './gradlew build -x test ` and when I want to run, I get the following error:

 cat test.nf 
Channel.fromPath("/home/lindenb/src/jvarkit-git/src/test/resources/toy.bam").
    splitSamples().
    dump()
$ ./nextflow run test.nf
N E X T F L O W  ~  version 19.08.0-SNAPSHOT
Launching `test.nf` [goofy_goldstine] - revision: f56eb762e4
Unknown method `splitSamples` on channel type

NF cannot find my operator. what am i missing in the code ?

Vũ Mạnh Tú

@glmanhtu

Hello guys. I'm trying to run nextflow with Kubernetes and have encountered some sort of problem with non-root user. Nextflow was trying to initialise a Pod but got an exception of "container init caused \"mkdir /mnt/tvu: permission denied". Below is the Kubernetes manifest generated by nextflow:

apiVersion: v1
kind: Pod
metadata:
  name: nf-68b883e7affa9dc5d6d5e721c75b21c4
  namespace: default
  labels: {app: nextflow, runName: sleepy-goodall, taskName: indexPeptides, processName: indexPeptides,
    sessionId: uuid-0e5b6c81-4527-4213-a9ba-16e40e424220}
spec:
  restartPolicy: Never
  containers:
  - name: nf-68b883e7affa9dc5d6d5e721c75b21c4
    image: omicsdi/crux:latest
    command: [/bin/bash, -ue, .command.run]
    workingDir: /mnt/tvu/work/68/b883e7affa9dc5d6d5e721c75b21c4
    volumeMounts:
    - {name: vol-1, mountPath: /mnt}
  securityContext: {runAsUser: 2801}
  volumes:
  - name: vol-1
    persistentVolumeClaim: {claimName: pride-pv-claim}

I have assigned rw permission to the persistent volume. When I removed the workingDir, the pod was started successfully and I was be able to cd into the /mnt/tvu/work/68/b883e7affa9dc5d6d5e721c75b21c4 folder. So, I suspect that the runAsUser and workingDir couldn't get along.
Is there anyone have experienced this issue?

Paolo Di Tommaso

@pditommaso

@lindenb try using ./launch.sh instead of nextflow script

then much simpler to test it using a test class

have a look at this example

https://github.com/nextflow-io/nextflow/blob/b49e6d74dae782def6be2b087d06fdd8fa6b9291/modules/nextflow/src/test/groovy/nextflow/splitter/TextSplitterTest.groovy#L31-L37

and the invocation as operator

https://github.com/nextflow-io/nextflow/blob/a8563a92af1b0e80c052951f2a5d563a512f5c0b/modules/nextflow/src/test/groovy/nextflow/extension/SplitTextOpTest.groovy#L13-L28

I would suggest also to open a draft pull request and keep a discuss thread there

Pierre Lindenbaum

@lindenb

@pditommaso :thumbsup:

Philip Jonsson

@kpjonsson

Question: In Nextflow version 19.07.0.5106, I get WARN: The channelcreatemethod is deprecated -- it will be removed in a future release when I use the Channel.create() method, but I find nowhere any mention of what's replacing it. Is the warning wrong or is there some undocumented feature I'm missing?

Ghost

@ghost~57581dbcc2f0db084a1ccd04

Don't worry, it won't happen overnight

you can start here for the conversation: https://gitter.im/nextflow-io/nextflow?at=5d47fec7e939ab2447fcc680

AFAIK you can safely already remove the stuff like ch_input = Channel.create()

Philip Jonsson

@kpjonsson

@MaxUlysse Got it, thanks.

Ghost

@ghost~57581dbcc2f0db084a1ccd04

And I do believe that was the important part:

to me the only use for create that makes sense is when used with choice

https://gitter.im/nextflow-io/nextflow?at=5d4802a1475c0a0feb021c1b

Paolo Di Tommaso

@pditommaso

this is just great, isn't it ?

https://twitter.com/yokofakun/status/1159468857934929922

Hugues Fontenelle

@huguesfontenelle

hi!
How do I read the documentation for an older release? ie 19.04

well I guess I can simply read in github ..

Paolo Di Tommaso

@pditommaso

indeed

evanbiederstedt

@evanbiederstedt

@pditommaso
this is just great, isn't it ?
https://twitter.com/yokofakun/status/1159468857934929922

Confirmed, this is great

evanbiederstedt

@evanbiederstedt

https://gitter.im/nextflow-io/nextflow?at=5d4802a1475c0a0feb021c1b

Could you give us a few examples @pditommaso to illustrate this point? It still feels a bit abstract without a few concrete examples

Pierre Lindenbaum

@lindenb

@evanbiederstedt @pditommaso thanks ! I'm still not convinced it's the right way to implement the idea. 1) one can still use a bash script to extract the data 2) or it would be better using a 'static' method: something like:Channel.fromMap("my.bam").flatMap(Htsjdk::extractSamples)

Taylor Falk

@taylor.f_gitlab

Is there an easy way to pass two channel outputs and some associated strings to a process? I'm currently trying output = Channel.from(['filter1', channel1_out], ['filter2', channel2_out]) but this only returns the blank DataflowQueue(queue=[]) string inside the process.

Stijn van Dongen

@micans

@taylor.f_gitlab that would mean you'd have channels in a channel. What are you trying to achieve? What's in channel1_out and channel2_out?

Taylor Falk

@taylor.f_gitlab

@micans those channels contain singular files, so I am really just trying to run the next process on each output from channel1 and channel2, and passing along the correct filter string. How do I move those files into one channel, concat?

Stijn van Dongen

@micans

@taylor.f_gitlab do you have a source process? Normally you'd see something like

output: set val('filter'), file('*.txt') into channel1

then if there are multiple files in that output that you want to flatten you can use transpose().

Taylor Falk

@taylor.f_gitlab

Oh that's a good idea, let me try it that way.

Gabriel Abud

@G_Abud_twitter

I'm getting the error: fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied when trying to run a pipeline on AWS Batch. Pretty sure it's due to the s3 access of the workDir but I've double checked and I am the owner of that bucket. Any ideas?

Combiz

@combiz_k_twitter

Hi, this may be more of a singularity question than a nextflow one. I'm trying to run a test hello world nextflow script with -with-singularity image.sif on an HPC cluster. There seems to be an issue with mounting as the short script runs but produces an error `Command error: Fatal error: cannot create 'R_TempDir'. Any ideas? My guess is the singularity container can't write files to the HPC filesystem? Thanks for any pointers.

Combiz

@combiz_k_twitter

Ok so the script now runs with containerOptions '-B $PWD:/tmp' in the process

Johannes Alneberg

@alneberg

A question regarding input of memory specifications. Using the config

params {
  memParam = 7.GB
}

process {
  withName:sayHello {
    memory = {2 * params.memParam}
  }
}

works fine when memParam is not used, but when it is: nextflow run main.nf -c base.conf --memParam 8.GB, I get either

ERROR ~ Error executing process > 'sayHello (3)'

Caused by:
  No signature of method: java.lang.Integer.multiply() is applicable for argument types: (java.lang.String) values: [8.GB]
Possible solutions: multiply(java.lang.Character), multiply(java.lang.Number)

Error executing process > 'sayHello (1)'

Caused by:
  Not a valid 'memory' value in process definition: 8.GB8.GB

depending on the order of the multiplication. What am I missing? (I'm on Nextflow version 19.04.1 build 5072)

Paolo Di Tommaso

@pditommaso

umm, because when passing on the command line the string 8.GB does not get parse to Memory unit object

therefore for the interpreter it's a string

Johannes Alneberg

@alneberg

Is it possible to go around?

Paolo Di Tommaso

@pditommaso

you may try 2 * (params.memParam as nextflow.util.MemUnit)

quite ugly

Johannes Alneberg

@alneberg

Well, rather ugly than broken, I'll give it a try

it throws me unable to resolve class nextflow.util.MemUnit

Alaa Badredine

@AlaaBadredine_twitter

not Util ?

Johannes Alneberg

@alneberg

Same error using Util I'm afraid

Where communities thrive

People

Repo info

Activity