These are chat archives for nextflow-io/nextflow

27th
Apr 2018
Andreas Wilm
@andreas-wilm
Apr 27 2018 01:45
Hi @pditommaso and all, quick question on config-profiles. If I wanted something to apply to all profiles (e.g. shell = ['/bin/bash', '-euo', 'pipefail']) what's the correct way to do it? Is the standard profile always loaded first and then overwritten by other profiles?
Paolo Di Tommaso
@pditommaso
Apr 27 2018 05:35
@andreas-wilm nope, standard is used only in no profile is specified
if you want to make it the default for in any profile set process.shell = etc outside the profiles scope
anybody in Boston area interested to this :point_up:
Andreas Wilm
@andreas-wilm
Apr 27 2018 05:53
Thanks @pditommaso That makes sense
Paolo Di Tommaso
@pditommaso
Apr 27 2018 05:54
nice
Simone Baffelli
@baffelli
Apr 27 2018 08:23
maybe nextflow log -F 'process =~ /polCov.*/' Indeed!
Vladimir Kiselev
@wikiselev
Apr 27 2018 10:11
Sorry, what is the command to clean cache/work folder? I’ve seen it before but can’t find now...
Evan Floden
@evanfloden
Apr 27 2018 10:12
nextflow clean
Vladimir Kiselev
@wikiselev
Apr 27 2018 10:12
Cool, thanks!
Rohan Shah
@rohanshah
Apr 27 2018 17:31
is there executor support for AWS ECS (as opposed to AWS Batch)? I remember seeing it somewhere but can no longer find it in the documentation
Paolo Di Tommaso
@pditommaso
Apr 27 2018 17:32
nope
what do you think would be the benefit ?
Rohan Shah
@rohanshah
Apr 27 2018 17:33
the startup time for AWS Batch is high especially for small/quick processes
startup time per job that is
Paolo Di Tommaso
@pditommaso
Apr 27 2018 17:34
that's true
Mike Smoot
@mes5k
Apr 27 2018 17:34
sounds like another use case for auto batching processes!
Paolo Di Tommaso
@pditommaso
Apr 27 2018 17:34
true !
Rohan Shah
@rohanshah
Apr 27 2018 17:34
willing to add it myself (an ecs executor) if there is any documentation or guidance for adding new executors
auto batching?
Paolo Di Tommaso
@pditommaso
Apr 27 2018 17:36
the source code IS the documentation :)
Rohan Shah
@rohanshah
Apr 27 2018 17:36
haha touché
Paolo Di Tommaso
@pditommaso
Apr 27 2018 17:36
but surely I can advice you if you are willing to do that
Rohan Shah
@rohanshah
Apr 27 2018 17:37
ok sounds good I will check my timeline soon and reach out, appreciate the help!!
also curious what auto batching is?
Paolo Di Tommaso
@pditommaso
Apr 27 2018 17:37
however, have you tried before the current NF ignite executor over Ec2 istances ?
Rohan Shah
@rohanshah
Apr 27 2018 17:39
i have not, however and unfortunately, the requirements I've been given require a more "managed" solution
Paolo Di Tommaso
@pditommaso
Apr 27 2018 17:40
I see, also what about EKS?
Rohan Shah
@rohanshah
Apr 27 2018 17:40
thats the other solution im considering
have you had good experience (quickness etc.) with K8s executors?
Mike Smoot
@mes5k
Apr 27 2018 17:41
auto batching is an idea that's been floated that would allow inputs to be batched in a way that a nextflow process would execute all of the inputs in a single unix process instead of spawning new processes each time. The use case is to address the "many small/fast processes" problem. Unfortunately, this is tricky to implement and I don't have the bandwidth to dig into it.
Paolo Di Tommaso
@pditommaso
Apr 27 2018 17:41
I still consider NF support for k8s experimental, but the latest round of improvements is very good IMO
thanks Mike, didn't want to avoid the question
IMO regarding K8s I'm very eager about user feedback
let me know if you try it
Rohan Shah
@rohanshah
Apr 27 2018 17:43
awesome, if I have the bandwidth to try it out I will definitely report back any findings
auto batching also seems very useful for us! cool to hear that is being thought about
Paolo Di Tommaso
@pditommaso
Apr 27 2018 17:44
I think it's a very promising platform around which there's big consensus, therefore makes sense to invest on it
true, batching could be useful, a bit painful to implement
may I ask what's your org ?
Rohan Shah
@rohanshah
Apr 27 2018 18:32
hey yes sorry its a company called Blackfynn
Paolo Di Tommaso
@pditommaso
Apr 27 2018 18:33
no pb, nice
boston area?
Rohan Shah
@rohanshah
Apr 27 2018 18:34
we are using nextflow to do some internal pre-processing on files imported into our platform as a sort of general, on-demand workflow engine
philadelphia actually
Paolo Di Tommaso
@pditommaso
Apr 27 2018 18:34
:+1:
cool
ah, neurological disease
Rohan Shah
@rohanshah
Apr 27 2018 18:36
yes things such as processing imaging and timeseries data
tbugfinder
@tbugfinder
Apr 27 2018 18:48
I've run nextflow on AWS Batch with reporting enabled. While processing I've terminated an instance, so those jobs got failed status and were restarted successfully. The report summary lists all failed and success jobs and doesn't honor the restart. In that sense I have to check all failed jobs, correct or did I miss a section ?
Paolo Di Tommaso
@pditommaso
Apr 27 2018 18:49
the restart of jobs you mean ?
I mean the report does not include the restarted jobs ?
tbugfinder
@tbugfinder
Apr 27 2018 19:27
Restarted jobs are included however even if finished successfully related failed jobs are still reported too, however I wouldn't care about these.
Paolo Di Tommaso
@pditommaso
Apr 27 2018 19:31
are you using errorStrategy = 'retry' ?
tbugfinder
@tbugfinder
Apr 27 2018 19:56
yes, two times retry, finally ignore
Paolo Di Tommaso
@pditommaso
Apr 27 2018 19:59
if you use errorStrategy = 'ignore' and maxRetries = 2 it should do the same, but the task retry should be delegated to Batch, therefore you should not see the failed one in the error
but make a test before .. just to be sure
/offline