These are chat archives for nextflow-io/nextflow

21st
Nov 2016
Denis Moreno
@Galithil
Nov 21 2016 11:44
Is there a plan to support Amazon's EBS in nextflow ?
Paolo Di Tommaso
@pditommaso
Nov 21 2016 13:04
@Galithil Should not be too difficult to add it. Could you open a feature request for that on GitHub?
Denis Moreno
@Galithil
Nov 21 2016 13:04
sure
Félix C. Morency
@fmorency
Nov 21 2016 14:39
@pditommaso ran my fist whole study using NF on multiple computers using built-in ignite support. everything worked as expected. thanks a lot
Paolo Di Tommaso
@pditommaso
Nov 21 2016 14:39
very nice
you may want to blog post about your use case, it would be very interesting to know more about that
Félix C. Morency
@fmorency
Nov 21 2016 14:45
interesting idea. i also made some NF publicity to riot games. one of the dev contacted me about luigi-related stuff. told him we were moving away from it
Paolo Di Tommaso
@pditommaso
Nov 21 2016 14:46
:smile:
Mokok
@Mokok
Nov 21 2016 15:12
I may be interested in your blog if you post it @fmorency ;)
Paolo Di Tommaso
@pditommaso
Nov 21 2016 15:12
I'm interested in yours as well @Mokok :)
Mokok
@Mokok
Nov 21 2016 15:17
haha, well, i haven't touched NF for a while now, i only keep reading your updates. We lost the "call for tenders" (?) but i'm now working on a similar project whose task scheduler is a bit messy... :D
Phil Ewels
@ewels
Nov 21 2016 16:20
Hello! Quick simple question: I have a process that can sometimes generate files, sometimes not. I need to create an output channel for them to be saved to the publishDir (right?), but then the process fails with an error if those files are not found.
How should I get around this?
Paolo Di Tommaso
@pditommaso
Nov 21 2016 16:54
if the output is optional you can declare it as ... optional
eg
process foo {
  output: 
  file 'bar' optional true
  :
}
Trevor Tanner
@tantrev
Nov 21 2016 17:02
Would it be difficult to add support for "memory" and "cpu" resources to NF's local executor?
Paolo Di Tommaso
@pditommaso
Nov 21 2016 17:03
it's already supported
Trevor Tanner
@tantrev
Nov 21 2016 17:08
Awesome. Does NF occasionally spawn a new process of itself then? I occasionally see several ".log" and ".trace" files generated after only running NF once.
Trevor Tanner
@tantrev
Nov 21 2016 17:14
I've just been seeing some weird behavior while trying to execute NF as a command through Google's [dockerflow] (https://github.com/googlegenomics/dockerflow) and [Genomics Pipeline] (https://cloud.google.com/genomics/reference/rest/v1alpha2/pipelines). The parent NF process seems to terminate without warning.
Paolo Di Tommaso
@pditommaso
Nov 21 2016 17:15
Does NF occasionally spawn a new process of itself then? I occasionally see several ".log"
No, it executes one log and trace for each run
Running NF via Dockerflow? you don't need Dockerflow to run NF in Google Cloud ..
Trevor Tanner
@tantrev
Nov 21 2016 17:19
Interesting, what would you recommend instead?
Paolo Di Tommaso
@pditommaso
Nov 21 2016 17:21
Though, it's not (yet) as well integrated as AWS. I guess the simplest way currently is via Elasticluster
Trevor Tanner
@tantrev
Nov 21 2016 17:25
Yeah, that's what I was doing before. Elasticluster's just wasteful when a few outlier tasks at the end of a job only use a small fraction of the cluster.
Paolo Di Tommaso
@pditommaso
Nov 21 2016 17:28
unfortunately NF currently supports elastic deployment only for the AWS
though should not be too complicated to add Google cloud
Trevor Tanner
@tantrev
Nov 21 2016 17:33
Interesting. Ya, that's basically what I've tried to do by marrying NF w/dockerflow. The only issue I've had seems to be this NF process thing. My ghetto solution has been to just poll every 60 seconds to see if NF is still running - http://pastebin.com/raw/iHJfxMUG
Paolo Di Tommaso
@pditommaso
Nov 21 2016 17:34
this is the launcher script you use the container ?
Trevor Tanner
@tantrev
Nov 21 2016 17:35
correct - it's the only way I've been able to get things to work
Paolo Di Tommaso
@pditommaso
Nov 21 2016 17:35
why are you launching NF in background ?
Trevor Tanner
@tantrev
Nov 21 2016 17:37
well what happens if I don't, is that docker for some reason thinks NF stops when it really hasn't. Hence, I've tried to work around it by launching it background and just monitoring to see if NF has actually terminated.
eventually thinks NF stops, it doesn't happen immediately
I've been running rather large jobs with thousands of files though, not sure if that's making a difference
Paolo Di Tommaso
@pditommaso
Nov 21 2016 17:39
weird
Mike Smoot
@mes5k
Nov 21 2016 20:28

Hi Paolo @pditommaso I just upgraded to 0.22.5 and I've been seeing this error

ERROR ~ No such variable: params.clean_up

when I run a pipeline without specifying the clean_up param on the command line. My code looks like this:

clean_up    = params.clean_up  ?: false

In past versions if I didn't set --clean_up then I'd just get the default false behavior. Now it seems like I must specify --clean_up. Is this expected?

Paolo Di Tommaso
@pditommaso
Nov 21 2016 20:30
Yes nextflow-io/nextflow#243
having not defined params and variables can be a problem in large scripts
just declare at the top of your script
params.clean_up = false
Mike Smoot
@mes5k
Nov 21 2016 20:34
Ok, I see the problem and why the change was added, however to return to the previous behavior with defaults I need to update a bunch of code...
Paolo Di Tommaso
@pditommaso
Nov 21 2016 20:34
why?
Mike Smoot
@mes5k
Nov 21 2016 20:37

This code no longer behaves the same way:

clean_up    = params.clean_up  ?: false

It will cause the error above rather than set clean_up to false.

Paolo Di Tommaso
@pditommaso
Nov 21 2016 20:37
params can be written only once, if you initialise at the beginning of the script there should not be any difference with
params.clean_up = false
clean_up  = params.clean_up
Mike Smoot
@mes5k
Nov 21 2016 20:38
Sure, that'll work - I just need to update all my code to do that.
Paolo Di Tommaso
@pditommaso
Nov 21 2016 20:43
your code will be more robust, put it this way ;)
Mike Smoot
@mes5k
Nov 21 2016 20:52
Possibly, but also more ambiguous. It's not at all obvious from reading this code that params.clean_up will get the value from the command line rather than the value that is explicitly set on the previous line. There's magic happening to get the command line value and magic always makes me nervous. Anyway, now I know!
Paolo Di Tommaso
@pditommaso
Nov 21 2016 20:54
the problem with unassigned values is that it's really easy to have a typo in a var name and you will never realise
you can maintain the conditional expression i.e.
clean_up = params.clean_up  ?: false
but params.clean_up must be initialised before using it
Mike Smoot
@mes5k
Nov 21 2016 20:58
Sure, I understand.