These are chat archives for nextflow-io/nextflow

12th
May 2019
Richard Janis Goldschmidt (Beckert)
@SuperFluffy
May 12 11:24
I want to run some long computations (simulatinos) on AWS and found nextflow.io. Right now I am trying to figure out how to deal with spot instance interruptions and resuming the computations. My computations write checkpoints every few minutes. If an interrupt happens, can nextflow help me resume them?
Right now I want to push the checkpoints to S3 and, upon resume, have some logic that figures out which checkpoint to pull from S3 to resume computation from there.
Rad Suchecki
@rsuchecki
May 12 11:42
I may be wrong @SuperFluffy but I think that checkpointing may not go well with nextflow.
NF takes care of resuming of a workflow, but if an individual process is interrupted then it will be re-computed
Can your computations be easily/naturally broken-up into separate processes?
Richard Janis Goldschmidt (Beckert)
@SuperFluffy
May 12 11:44
@rsuchecki Well, I suppose I could model each simulation as a series of successive steps.
Will think about that. Maybe this is a natural way of doing things.
Rad Suchecki
@rsuchecki
May 12 11:48
If this model fits the kind of computational problem(s) you deal with then NF is really worth trying
Richard Janis Goldschmidt (Beckert)
@SuperFluffy
May 12 11:49
I can definitely re-interpret my computational problem to fit this model. :-D
Rad Suchecki
@rsuchecki
May 12 11:49
:v:
Laurence E. Bernstein
@lebernstein
May 12 18:53
@micans When I run Nextflow it says there is no -w option and it is not listed under [options].