These are chat archives for nextflow-io/nextflow

30th
Jan 2019
Hugues Fontenelle
@huguesfontenelle
Jan 30 16:22

I was trying to use Apache Ignite, without success, until I read some old conversation in this channel.

ok, ignite can work as a queue manager but it's not a replacement for a batch scheduler
what I want to say that if you are planning to have stable usage of NF, I would suggest to install something like slurm

Reading the executor documentation, I had made the same assumption than some other people: that I could run nextflow node on one computer, and nextflow run tutorial.nf -process.executor ignite on another..

Case usage: I have access to several servers that are often idle, my target cluster (SLURM) is too cumbersome to use (two-layer auth, no ssh, no copy-paste), I'd like to have a development environment a bit more similar to a cluster than the local executor, ie set cpus/memory/time process directives.

Can I use Ignite as a queue manager?
So far I've tried multicast, IP mode and S3 bucket, but either way the pipeline always runs locally :(

Paolo Di Tommaso
@pditommaso
Jan 30 16:24
no cloud ?
Hugues Fontenelle
@huguesfontenelle
Jan 30 16:28

That will be the question for next month, yes :)
I'm looking forward trying GCP!

But I like the idea of scavenging some cycles from the oft-idle server farm ..

Paolo Di Tommaso
@pditommaso
Jan 30 16:29
if you have batch scheduler, you can deploy it as a MPI workload
Hugues Fontenelle
@huguesfontenelle
Jan 30 16:30
yep yep I read. I don't have a scheduler
Paolo Di Tommaso
@pditommaso
Jan 30 16:30
otherwise you need to tear up and down the daemons on each node
I won't suggest it
Hugues Fontenelle
@huguesfontenelle
Jan 30 16:49
ok. ty
Yasset Perez-Riverol
@ypriverol
Jan 30 16:52
Hi all, do we have a macanism in nextflow to reuse workflows, let said I have workflow A and I want a workflow B that continues from workflow A.
Paolo Di Tommaso
@pditommaso
Jan 30 16:56
#!/bin/bash
nextflow run A 
nextflow run B -resume --some-output-files-from-A
Hugues Fontenelle
@huguesfontenelle
Jan 30 17:00
From workflow A, publishDir some files. From workflow B, assume the files are in the correct location. Use set -e in @pditommaso 's bash script :-)
That means both workflows share the same workdir and work/ folder.
Yasset Perez-Riverol
@ypriverol
Jan 30 17:04
Probably this is more a question for nf-core. It would be great like to have like a layer workflow, that I can said in B in the head something like extends A and I don’t need to care to mantain A and both workflows have their own lifecycle
This is goog feature when you have communities workflows and you want to reuse the workflow of someone else wihout writing A yourself
But this is probably a bad idea, I just came across that after Ctrl-C Ctrl-V a couple of times.
Paolo Di Tommaso
@pditommaso
Jan 30 17:06
Yasset Perez-Riverol
@ypriverol
Jan 30 17:07
thanks a lot !!!!
Paolo Di Tommaso
@pditommaso
Jan 30 17:07
:satisfied:
Maxime Garcia
@MaxUlysse
Jan 30 18:46
$prefix is not defined
I'm guessing that's a good start
Also why do you have this _ after the {} ?
lastwon1216
@lastwon1216
Jan 30 18:50
oh let me remove that and try again
thank you
Maxime Garcia
@MaxUlysse
Jan 30 19:05
I'm sorry, your script is a little too long to catch up what is working and what is not
You should try to remove as much as possible
like executor, tag, publishDir...
Maxime Garcia
@MaxUlysse
Jan 30 19:10
But still, this variable prefix is not defined anywhere
lastwon1216
@lastwon1216
Jan 30 19:32
i got to define prefix as prefix="sed 's/.fastq//' ${params.reads}"
lastwon1216
@lastwon1216
Jan 30 19:57
command I used is:
nextflow run works2.nf --reads 'test{2,3}.fastq.gz' --genome reference_genome --prefix test
Maxime Garcia
@MaxUlysse
Jan 30 20:05
Sorry not on my computer anymore. I'll try out first thing tomorrow morning. I'm pretty sure it should be easy to debug
Alexey Dushen
@blacky0x0
Jan 30 20:55
Hi there. Is it possible to debug with any IDE the *.nf DSL script?
lastwon1216
@lastwon1216
Jan 30 23:43
@MaxUlysse thank you for your help!