These are chat archives for nextflow-io/nextflow

7th
Feb 2019
Alexey Dushen
@blacky0x0
Feb 07 11:11
Hi there, I've just noticed that parallel processes are drawn with Graphviz as a single item.
Is it possible to draw as much processes as parallel processes I have?
Paolo Di Tommaso
@pditommaso
Feb 07 11:34
nope
Alexey Dushen
@blacky0x0
Feb 07 11:48
it's a pity. ok, got it
Paolo Di Tommaso
@pditommaso
Feb 07 11:49
most of time, there are son many instances that's just useful
anyhow you may want to open a feature request on the GitHub project repo
micans
@micans
Feb 07 13:04
Could print the number of processes somewhere.
Paolo Di Tommaso
@pditommaso
Feb 07 13:05
question or assertion ?
micans
@micans
Feb 07 13:06
suggestion.
Paolo Di Tommaso
@pditommaso
Feb 07 13:09
Like this ?
Screenshot 2019-02-07 at 14.08.55.png
Maxime Borry
@maxibor
Feb 07 15:04

Dear all,
Can I use symlinks as inputs for a NF-pipeline, and if so, how ?
My pattern for input reads works fine with files, but not with symlinks...

nf.conf

reads = "/path/to/*_R{1,2}_001.fastq.gz"
With symlinks, it gives me
ERROR ~ Cannot find any reads matching: /path/to/*_R{1,2}_001.fastq.gz
Alexander Peltzer
@apeltzer
Feb 07 15:09
As NF typically symlinks files itself to the workdir you use, might require something like: https://www.nextflow.io/docs/latest/process.html#stageinmode
Never tried it though myself, but could be that symlinks of symlinks don't work , never gave it a thought before :question:
Maxime Borry
@maxibor
Feb 07 15:17
Well, there is a (default to true) parameter for channels called followLinks so it should work, no ?
https://www.nextflow.io/docs/latest/channel.html#fromfilepairs
Paolo Di Tommaso
@pditommaso
Feb 07 15:17
what if you set explicitly ?
Alexander Peltzer
@apeltzer
Feb 07 15:17
True
Maxime Borry
@maxibor
Feb 07 15:19

so

params.reads = /pat/to/*_R{1,2}_001.fastq.gz"

Channel
    .fromFilePairs( params.reads, followLinks = true, size: params.pairedEnd ? 2 : 1,  )
    .ifEmpty { exit 1, "Cannot find any reads matching: ${params.reads}\n" }
    .set {reads_to_trim}
Paolo Di Tommaso
@pditommaso
Feb 07 15:19
no
followLinks: true
Maxime Borry
@maxibor
Feb 07 15:20
True (excuse my python). Sorry true
Well, unfortunately still
ERROR ~ Cannot find any reads matching: /path/to/*_R{1,2}_001.fastq.gz
(I only have symlinked files in this directory)
(NF version 18.10.1)
Paolo Di Tommaso
@pditommaso
Feb 07 15:23
you may want open an issue on GH including a reproducible test case
Maxime Borry
@maxibor
Feb 07 15:23
ok
Maxime Borry
@maxibor
Feb 07 15:46
...Nextflow is fine (how surprising)
Someone modified our data directories and all the symlinks were therefore broken :expressionless:
Paolo Di Tommaso
@pditommaso
Feb 07 15:47
:wink:
Maxime Garcia
@MaxUlysse
Feb 07 15:47
Scary
Alexander Peltzer
@apeltzer
Feb 07 16:03
SHH IT <3
;-)
micans
@micans
Feb 07 16:05
@pditommaso apologies for brevity; I was commenting on drawn with Graphviz (multiple parallel processes). I interpreted it as drawing multiple arcs, and suggested to just put a number on the arc of the process ... not sure I interpreted the request correctly.
Paolo Di Tommaso
@pditommaso
Feb 07 16:07
ah, maybe it would be nice to have the full graph expansion, just to have useless beautiful big graphs :sunglasses:
micans
@micans
Feb 07 16:08
I know all about them .... bioinformaticians love their hairballs and their circos plots :grimacing:
Paolo Di Tommaso
@pditommaso
Feb 07 16:08
ahah
Michael L Heuer
@heuermh
Feb 07 16:21
@pditommaso @micans Let me know if you'd like an interactive graph in Cytoscape. I've been meaning to bring all my Cytoscape Apps up-to-date sometime soon, and when I get into that, I may get on a roll and write some new stuff
Paolo Di Tommaso
@pditommaso
Feb 07 16:23
thanks, tho main change would be in NF code
@heuermh planning to join BOSC this summer in Basel ?
Michael L Heuer
@heuermh
Feb 07 16:25
No problem, I can write Groovy. It's just like Scala, right? ;)
Yeah I got travel approval, looking forward to it!
Paolo Di Tommaso
@pditommaso
Feb 07 16:26
great for both ! :smile:
Bede Constantinides
@bede
Feb 07 17:04
New user here: .command.run: line 57: activate: No such file or directory likely pointing to a conda env issue?
Paolo Di Tommaso
@pditommaso
Feb 07 17:05
sounds like that
Bede Constantinides
@bede
Feb 07 17:09
Hmm, thanks. It wasn't doing it earlier, and I've changed nothing about my env.yml 🤔
Paolo Di Tommaso
@pditommaso
Feb 07 17:10
earlier when ?
Bede Constantinides
@bede
Feb 07 17:11
Oh, it just started throwing that error a few minutes ago. I've since deleted the work dir and hence rebuilt the conda env from the same yaml and it's not working. I've probably changed something. Thanks!
Paolo Di Tommaso
@pditommaso
Feb 07 17:13
change in the tsk work dir and run bash .command.run and see what's is failing
Bede Constantinides
@bede
Feb 07 17:32
Thanks. Yes the source activate line was failing. I think my shell had picked up some weird state… Running in a new shell works fine. Sorted!
Paolo Di Tommaso
@pditommaso
Feb 07 17:32
:+1:
Stephen Kelly
@stevekm
Feb 07 18:53
is there a way to set a time limit for how long a task can be in queue? For example, Nextflow submits a job to SLURM, then the job sits in the SLURM queue for an inordinate amount of time e.g. 12hrs, I would prefer to cancel the job and try to re-submit it after waiting for 4 hours. Is it possible to get Nextflow to do this?
I put time = "4h" as a process directive but that is only the amount of time that a job is allowed to run for. I want something like queueTime which is the amount of time allowed to wait for the job to start running