These are chat archives for nextflow-io/nextflow

13th
Jul 2018
Raoul J.P. Bonnal
@helios
Jul 13 2018 04:34
@Lenbok I let NF create the job definition. Then I did a revision adding the volume mapping and the bind will be available in your Docker. A tip, as ENV variable define the TMPDIR to your EFS so that the software that requires temp file will use a directory of the host. As reported in the NF docs the ami has a limited size so using the default tmp can be an issue. On the NF side, configure the container to point to the jobdefinition:// not to the docker image (please check the NF doc, there is a comment about the jobdefinition). If you are using spot , check that your bid is enough to start the instances otherwise they will remain runnable forever.
Len Trigg
@Lenbok
Jul 13 2018 05:04
@helios Thanks, that's helpful!
Alan B. Christie
@alanbchristie
Jul 13 2018 05:44
I suspect my collapse of processing throughput may have something to do with task velocity (for want of better term). If Nextflow itself is thread limited then there will come a point when there's just too much churn starting and stopping individual tasks (the velocity or rate of change of task is too high). If my assumption's right and there's only one thread there will come a point when Nextflow cannot manage the startup of any more tasks? I have 500 cores, 1.6 million tasks to run and the average task length is 47 seconds with a minimum task length of 5 seconds. Could Nextflow be overloaded just starting and stopping tasks? Is there a limit to task velocity? I'm just guessing here, but it is the maxed-out java process that got me worrying.
That might explain why my 144 core test runs and all my benchmarking was successful but, with 500 cores I breached a task limit - and there there will be a limit if the controller only has access to one thread. Anyway, just supposition at the moment - @pditommaso what do you think?
Francesco Strozzi
@fstrozzi
Jul 13 2018 07:42
@Lenbok I am curious about which performance do you see having the big genomic indexes on an EFS volume instead of doing a local stage of the indexes on the EBS volume attached to the AMI.
the only downside of local staging is of course multiple copies of the same index on the AMI when you run multiple jobs on it. But in terms of performance EBS should be better I think and less expensive than EFS……just asking, as I don’t have a direct experience using EFS
@helios you seems to have gone quite far in setting up the EFS binding on Docker for NF and Batch. Would you be interested in updating the NF docs and adding these tips and instructions ? I think it will be interesting for the users to have multiple options to access reference data for their analysis on AWS Batch
Raoul J.P. Bonnal
@helios
Jul 13 2018 07:53
@fstrozzi sure, I am happy to contribute and I'll try to update the docs in the upcoming days.
Francesco Strozzi
@fstrozzi
Jul 13 2018 07:55
:+1:
thanks
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:00
@alanbchristie hard to say, the NF process should not use all CPU time, there could something going wrong with the garbage collector
Alan B. Christie
@alanbchristie
Jul 13 2018 08:01
@pditommaso There was a lot of churn in Memory recycling.
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:01
you should try to profile the execution with Flight record or YourKit or similar
Alan B. Christie
@alanbchristie
Jul 13 2018 08:02
I was going to restart the cluster with 144 cores, that seemed to behave itself, and then slowly add nodes.
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:04
are you using autoscaling feature ?
Alan B. Christie
@alanbchristie
Jul 13 2018 08:04
The jobs are short but a job every 90mS (45/504) might be too fast.
No, just adding machines using a shared directory on EFS.
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:05
4.5/504 what unit ?
Alan B. Christie
@alanbchristie
Jul 13 2018 08:05
Sorry, typo.
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:05
90 milliseconds ?
Alan B. Christie
@alanbchristie
Jul 13 2018 08:05
Think so.
And that will continue for about 30 hours.
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:07
using a docker ?
Alan B. Christie
@alanbchristie
Jul 13 2018 08:07
in-docker, yep
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:07
not sure to suggest NF for this granularity
Caspar
@caspargross
Jul 13 2018 08:08
Is there a file.baseDir command similar to file.baseName ? I cant find it in the documentation. Should return the name of the parent directory.
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:08
only the creation of the work-dir, script wrapper and container add a much longer overhead
@caspargross file.parent
Alan B. Christie
@alanbchristie
Jul 13 2018 08:09
Agreed.
Caspar
@caspargross
Jul 13 2018 08:09
:thumbsup:
Alan B. Christie
@alanbchristie
Jul 13 2018 08:09
So I guess the first hour's fine (all cores at full capacity) but the sustained throughout is maybe too much.
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:11
this looks a different problem, you should profile JVM execution to see what's slowing down, I suspect a memory leak or something similar
Alan B. Christie
@alanbchristie
Jul 13 2018 08:11
Understand, do you have rough benchmarks for Docker workflows like this? Or is there something that can be logged to display a growing "backlog"?
For the time-being I think I'll grow the system in smaller steps rather than just throw 500 cores at the problem.
@pditommaso Thanks by the way. It'll be an interesting exercise.
Thanks by the way. It'll be an interesting exercise
yes, if you identify the cause even better :smile:
Alan B. Christie
@alanbchristie
Jul 13 2018 08:14
We'll see. Thanks.
Francesco Strozzi
@fstrozzi
Jul 13 2018 08:14
1.6M jobs is really large….couldn’t also be some bottleneck with Ignite ?
Alan B. Christie
@alanbchristie
Jul 13 2018 08:16
@fstrozzi I think we agree that we've moved out of Nextflow's comfort-zone.
Francesco Strozzi
@fstrozzi
Jul 13 2018 08:16
yeah, but is outside the comfort-zone that the most interesting things happen :smile:
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:17
Ignite executor is not battle tested, but I've used NF with 3-5 millions jobs with no problem
Francesco Strozzi
@fstrozzi
Jul 13 2018 08:17
with Ignite as backend ?
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:18
yes
Alan B. Christie
@alanbchristie
Jul 13 2018 08:22
It's unlikely to be the number of jobs as @pditommaso says. I suspect it's the task velocity. I suspect that if my jobs lasted longer (on average) then we'd be OK. The master's on one thread (?) and has to manage a new task every 90mS and it's clearly maxed out its own core. Perfectly reasonable.
Paolo Di Tommaso
@pditommaso
Jul 13 2018 08:26
but the queueSize limit should backstop the submission
Alan B. Christie
@alanbchristie
Jul 13 2018 08:28
OK, the queueSize was set to 600. According to the log, Nextflow was maintaing this but the job throughput had dropped significantly.
It's a much smaller test but everything seemed to be OK on 144 cores for about 2.5 hours.
Alan B. Christie
@alanbchristie
Jul 13 2018 09:06
OK, using -resume on 3 new nodes (having cleaned out the old cluster path). 216 cores available and recognised by the master. Let's see if the throughput can be maintained...
java's %CPU seems to have settled down to around 35%
micans
@micans
Jul 13 2018 09:33
@pditommaso -- I have a question now about the earlier code I posted, about the last process, sample_reconstitute. (A) Is it the case that NF has to wait for all earlier processes sample_parallel (which I should have called sample_subparallel to finish, in order to group together the tuples for sample_reconstitute (as I assume it does not track tuple size). (B) If that is the case, would it be possible to have something that encapsulates ch_onion_middle.map { key, file -> return tuple(key.toString(), file) } .groupTuple().set { ch_onion_tip } into a single NF primitive, say emitTuple, so that NF can track tuple size?
Ouch, missing right parenthesis. Apologies for all resulting brain injuries
Paolo Di Tommaso
@pditommaso
Jul 13 2018 09:38
I really want to help, but you need to help me writing shorter questions
micans
@micans
Jul 13 2018 09:39
How about (A)? It does not necessarily need context. If I use groupTuple(), does NF need to wait for all the potential processes that can contribute to finish?
I appreciate short questions are best, but I do try to make my questions comprehensive, so that all the information is there ...
Paolo Di Tommaso
@pditommaso
Jul 13 2018 09:41
this is a chat, short questions, short answers :wink:
regarding a, it wait for all unless you specify size parameter
micans
@micans
Jul 13 2018 09:43
ah, that sounds cool, as if I can bundle the size tracking along. Thanks, should have looked better
Alan B. Christie
@alanbchristie
Jul 13 2018 09:48
The process on 216 cores looked OK to start with but after half an hour into the workflow the java task is regularly maxing out at over 100%. We've seen a way to make our tasks run for longer so I'm going to adjust the workflow and make sure that the jobs take much longer (15 to 30 minutes rather than 1).
Paolo Di Tommaso
@pditommaso
Jul 13 2018 09:49
makes sense
Michael Epstein
@miepstei
Jul 13 2018 10:05
Hi, apologies for a newbie question in advance...can multiple shell:''' ... ''' blocks be run sequentially within the same process? The use case is that I want to do some bash scripting then be able to run an R script in the same process. I can achieve this using Script -e '...' but I have to escape characters which is a bit tortuous
Paolo Di Tommaso
@pditommaso
Jul 13 2018 10:06
best way to do that is to have the R script as separate files
Michael Epstein
@miepstei
Jul 13 2018 10:07
ok - thanks very much
Paolo Di Tommaso
@pditommaso
Jul 13 2018 10:07
than you can combine as any other command
Michael Epstein
@miepstei
Jul 13 2018 10:09
thanks - keep up the good work.
micans
@micans
Jul 13 2018 10:26
I don't think size: works for me, as it's one-size fits all. In my case I can have (a) creates 3 subsamples, (b) creates 2 subsamples, (c) creates 8 subsamples. I flatten those 3+2+8=13 subsamples then later I can groupTuple them, but I can't pass those individual sizes along. Which would lead to the earlier question (B). Is there a suitable non-real-time place for asking that question?
Paolo Di Tommaso
@pditommaso
Jul 13 2018 10:38
There's a Google discuss group
micans
@micans
Jul 13 2018 10:40
OK, will ask/discuss there. Either I miss something or I think it could potentially be a cool feature if it fits in the design.
Paolo Di Tommaso
@pditommaso
Jul 13 2018 10:42
Even better, open an issue and described
Luca Cozzuto
@lucacozzuto
Jul 13 2018 10:42
Hi @pditommaso. There is a way to print something without caring about escaping characters using log.info?
Paolo Di Tommaso
@pditommaso
Jul 13 2018 10:42
*describe your use case
Luca Cozzuto
@lucacozzuto
Jul 13 2018 10:43
log.info """
\\\////
"""
micans
@micans
Jul 13 2018 10:43
Alright sounds great. I'm convinced I have a great use case :-)
Paolo Di Tommaso
@pditommaso
Jul 13 2018 10:43
Pipeline ASCII art?
Luca Cozzuto
@lucacozzuto
Jul 13 2018 10:43
yes
Paolo Di Tommaso
@pditommaso
Jul 13 2018 10:44
:facepalm:
Luca Cozzuto
@lucacozzuto
Jul 13 2018 10:44
joke
Paolo Di Tommaso
@pditommaso
Jul 13 2018 10:44
Google for groovy dollar-slashy strings
Luca Cozzuto
@lucacozzuto
Jul 13 2018 10:45
nope
if I have to escape I'll give up
Phil Ewels
@ewels
Jul 13 2018 11:06
I added ASCII art to the nf-core pipelines 😂
(And yes, I had to add a bunch of escape characters)
Sven F.
@sven1103
Jul 13 2018 11:14
@lucacozzuto
test = $/"""\\\////
////
"""/$
println test
this should work
as @pditommaso said
Luca Cozzuto
@lucacozzuto
Jul 13 2018 11:17
Screen Shot 2018-07-13 at 13.15.47.png
😂😂😂😂😂😂😂😂😂😂😂😂😂
Sven F.
@sven1103
Jul 13 2018 11:20
:+1:
Sven F.
@sven1103
Jul 13 2018 11:36
:joy:
marchoeppner
@marchoeppner
Jul 13 2018 12:22
hi - I was gonna try the new conda directive for software provisoning; quick question about that:
$runMultiQCFastq { conda = [ 'multiqc=1.5' ] time = { 2.h * task.attempt } }
would this work? getting an error that the executable wasn't found - guess this is wrong?
this would be in my config file, of course
miniconda2 is installed
nextflow versoin 0.30.2
Paolo Di Tommaso
@pditommaso
Jul 13 2018 12:24
nearly
list is not used for conda, and semicolon is required if they are one the same line ie.
marchoeppner
@marchoeppner
Jul 13 2018 12:25
yes, this was a pasting error, its across multiple lines in the code
Paolo Di Tommaso
@pditommaso
Jul 13 2018 12:25
process {
  withName:runMultiQCFastq { 
     conda = 'multiqc=1.5'  
    time = { 2.h * task.attempt } 
   }
}
marchoeppner
@marchoeppner
Jul 13 2018 12:26
great, thanks!
Paolo Di Tommaso
@pditommaso
Jul 13 2018 12:26
:+1:
Alan B. Christie
@alanbchristie
Jul 13 2018 13:57
After shuffling the processes around (thanks Tim) we've reduced the number of tasks (now about 20,000) but they all take much longer. The 504 cores are now operating at near 100% efficiency and the Nexflow java process is idling at around 3.4%. Phew! So, about 24 hours to go...
Raoul J.P. Bonnal
@helios
Jul 13 2018 13:58
@lucacozzuto ehehe I'm doing the same thing
tbugfinder
@tbugfinder
Jul 13 2018 14:11
Sidenote in regards to AWS EFS: Sine yesterday EFS supports provisioned throughout so we are no longer dependent on the size of the FS and also keep in mind that there a different types of EBS volumes.
Alexander Peltzer
@apeltzer
Jul 13 2018 14:14
Cool!
Paolo Di Tommaso
@pditommaso
Jul 13 2018 15:24
@tbugfinder how are using it with batch and NF ?
tbugfinder
@tbugfinder
Jul 13 2018 15:56
@pditommaso Right now we just mount it using custom AMI and use it as a shared directory (executables, tens of thousands of input files+final outputs) for processing. The S3 workdir setup is overhead but mandatory. Even encryption in transit and at rest is included (transparent).
Paolo Di Tommaso
@pditommaso
Jul 13 2018 16:00
How is compared to S3
tbugfinder
@tbugfinder
Jul 13 2018 16:47
S3 is object storage and files are copied to the node locally or EFS, processed, uploaded. EFS (Amazon NFS) is classic shared FS. Certainly there are unique features on top of S3 like using events on actions.