These are chat archives for nextflow-io/nextflow

20th
Mar 2017
Maxime Garcia
@MaxUlysse
Mar 20 2017 09:30
Hello @pditommaso we have two pipelines with common steps, is it possible yet to module this up or should we make two pipelines ?
Didn't really find anything in the docs about it, and I remember some talks here about this kind of questions
Paolo Di Tommaso
@pditommaso
Mar 20 2017 09:43
if they are sequential commands, you can put together in an external script or template
if you are referring sub-workflows they are not supported
Maxime Garcia
@MaxUlysse
Mar 20 2017 09:48
I was thinking more sub-workflows, but I'll check the templates
Thanks
Paolo Di Tommaso
@pditommaso
Mar 20 2017 10:03
welcome
Tim Diels
@timdiels
Mar 20 2017 16:02
https://www.nextflow.io/docs/latest/process.html#output-set-of-values
suggests that Channel.fromPath lists paths in a deterministic fashion. If so, what order are paths returned in?
Paolo Di Tommaso
@pditommaso
Mar 20 2017 16:03
umm .. where ? :)
Tim Diels
@timdiels
Mar 20 2017 16:03
It's implied, the example wouldn't work if the order was random
sometimes cow.fasta (for example) would be paired with human, sometimes with cow, sometimes with horse
Paolo Di Tommaso
@pditommaso
Mar 20 2017 16:05
umm .. maybe I need to change that example
items provided Channel.fromPath are not supposed to ordered
Tim Diels
@timdiels
Mar 20 2017 16:06
I'm nitpicking perhaps, but wanted to make sure
Ok, thanks
Paolo Di Tommaso
@pditommaso
Mar 20 2017 16:06
you are right that example could suggest that, I will modify it
Tim Diels
@timdiels
Mar 20 2017 16:18

https://www.nextflow.io/docs/latest/process.html#beforescript
Is this equivalent to

process foo {
  """
  source /cluster/bin/setup && echo bar
  """
}

?

Paolo Di Tommaso
@pditommaso
Mar 20 2017 16:18
yes
Tim Diels
@timdiels
Mar 20 2017 16:24
Ok, I wondered as I didn't understand the need for it, but I guess it's useful if the script section is conditional where you would have to copy paste the before script in each if-block otherwise
Paolo Di Tommaso
@pditommaso
Mar 20 2017 16:25
well, it can be useful to externalise the setting for a specific setting in the config file
don't forget you can put all process directives in the nextflow config file e.g.
process.beforeScript = 'this && that'
and you can use to configure a particular env
Tim Diels
@timdiels
Mar 20 2017 16:36
ah yes
Tim Diels
@timdiels
Mar 20 2017 17:20
How do you configure to use no more than 100 CPUs of an SGE cluster?
Paolo Di Tommaso
@pditommaso
Mar 20 2017 17:21
overall ?
Tim Diels
@timdiels
Mar 20 2017 17:21
Yes, for all processes
It's a shared cluster, I don't want to hog it entirely
Paolo Di Tommaso
@pditommaso
Mar 20 2017 17:22
you can limit the number of jobs queued not the cpus
see queueSize here
Tim Diels
@timdiels
Mar 20 2017 17:25
Ok, that may work for now; maybe IT can configure a limit through SGE
Paolo Di Tommaso
@pditommaso
Mar 20 2017 17:26
well in my experience the problem is to not overload the cluster with too many jobs
Tim Diels
@timdiels
Mar 20 2017 17:30
I believe ours is a FIFO queue, so we leave a good deal of cores idle in case others want to use the cluster; which is inefficient, we should fix that
Roman Valls Guimera
@brainstorm
Mar 20 2017 20:52
Hi there! Following on https://twitter.com/nextflowio/status/843895815110311936, $baseDir/data does not refer to /home/ubuntu/.nextflow/assets/brainstorm/bwa-nextflow-benchmark/data which is what I seek on /home/ubuntu/.nextflow/assets/brainstorm/bwa-nextflow-benchmark :-S
Ahaaa, got the problem now, reruns on the node/cluster for a shared pipeline do not re-pull the remote shared pipeline!: ./nextflow run brainstorm/bwa-nextflow-benchmark -with-timeline timeline.html -with-trace -with-dag flowchart.png
Roman Valls Guimera
@brainstorm
Mar 20 2017 20:57
I had to run ./nextflow drop brainstorm/bwa-nextflow-benchmark for it to incorporate the latest changes!
Paolo Di Tommaso
@pditommaso
Mar 20 2017 20:57
um.. your home is not on a shared dir ?
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:10
nope, I'm trying to run everything on a single instance, no shared filesystem
on AWS
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:11
have you figured out the problem ?
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:12
Yes: I had to run ./nextflow drop brainstorm/bwa-nextflow-benchmark for it to incorporate the latest changes!
nextflow drop and then nextflow run again
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:12
ahh, yes
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:12
Otherwise, nextflow does not detect the changes in the remote repo
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:13
you can also do nextflow pull .. etc
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:13
It should automatically by default on every run, imho, just a quick check on HEAD
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:14
umm, not sure
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:14
Shouldn't be that expensive and avoids confusion, I think
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:15
imagine you are using the code of somebody else
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:16
A fork?
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:16
you would run an updated version without being notified
not necessarily a fork
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:16
INFO: Pulling latest version with hash 9328042asdfa3
Ah, I see what you mean now
Would clash with running specific revisions perhaps
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:17
also
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:17
gotcha
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:17
anyhow you can force the latest change with nextflow run -latest <name> as well
you can replace this with
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:18
Sweet! Last but not least, what's the preferred way of adding system dependencies pre-run (apt-get install...) ... or should I resort to ansible?
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:18
params.genome = "$baseDir/data/seq/hg19.fa"
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:18
Yes, now I'll definitely will
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:19
we manage all pipeline deps with a container
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:19
Yes, but I unfortunately have to live in the past for certain environments (HPC)
I use docker on my own, but you know the drill :/
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:20
never heard about singularity ..? :)
otherwise you can use Environment module
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:25
the environment module to install system packages? oh, shell out an env variable that? Hm, bit hackish
And yeah, singularity, heard good things about it, still have to give it a spin
Also, is it possible to run nextflow run brainstorm/bwa-nextflow-benchmark -with-timeline timeline.html -with-trace -with-dag flowchart.png -profile cloud locally yet executing the stuff in the newly instantiated AWS cluster?
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:30
nope, you need to ssh into the AWS cluster
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:30
That is, some short of shortcut for ssh -c nextflow run <REPO> flags ubuntu@aws_instance.com
Gotcha
Really cool framework and super grateful for your support, enjoyed my first run ;)
When I saw Kubernetes was cooking already, got me even more hooked
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:31
you are really welcome !
let's expand Sweden NF community ;)
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:33
Well, you have strong advocates @scilifelab already, it's healthy and growing... plus I'll be moving to Australia soon anyway, but happy to play with it further there... provided that it also supports CWL sooner or later ;)
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:34
if you want contribute testing the support for Kubernetes you are welcome
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:34
I might try tomorrow actually, I have a k8s cluster instantiated with kops from a few days ago
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:35
ohoh, are you joining this project ?
Roman Valls Guimera
@brainstorm
Mar 20 2017 21:35
I am in it already ;)
Paolo Di Tommaso
@pditommaso
Mar 20 2017 21:35
:clap: :clap: :clap:
hope we will work on that
badribio
@sridhar0605
Mar 20 2017 22:17
Hello everyone,