These are chat archives for nextflow-io/nextflow

14th
Nov 2016
Denis Moreno
@Galithil
Nov 14 2016 12:44
Is it possible to use Amazon EC2 instances with nextflow without using EFS ? I'm a bit confused (that happens often)
Paolo Di Tommaso
@pditommaso
Nov 14 2016 12:45
yes, you will need to use a S3 bucket as work dir
nextflow run foo -work s3://bucket/..
Denis Moreno
@Galithil
Nov 14 2016 12:47
ookay. And I configure it through the aws scope
Paolo Di Tommaso
@pditommaso
Nov 14 2016 12:47
what do u mean?
This message was deleted
@jbyars I'm checking if I can improve it
Denis Moreno
@Galithil
Nov 14 2016 12:53
I'm trying to figure out how to configure it, it's just that I have docs to read
Paolo Di Tommaso
@pditommaso
Nov 14 2016 12:54
as described here
Denis Moreno
@Galithil
Nov 14 2016 12:55
yep that's what I'm reading
Paolo Di Tommaso
@pditommaso
Nov 14 2016 12:55
but when you will launch the pipeline specify the work dir over an existing bucket
Denis Moreno
@Galithil
Nov 14 2016 12:56
all the input files are copied to the work dir anyway, right ?
Paolo Di Tommaso
@pditommaso
Nov 14 2016 12:57
yes
Denis Moreno
@Galithil
Nov 14 2016 12:57
thanks
Paolo Di Tommaso
@pditommaso
Nov 14 2016 12:57
welcome
Phil Ewels
@ewels
Nov 14 2016 14:40
Hi @pditommaso - we're getting the error Cannot invoke method complete() on null object on a process. Any idea what it could mean?
The process seems to run, create the correct output, the command looks fine, the exitcode is fine, everything I can think of looks as expected
Paolo Di Tommaso
@pditommaso
Nov 14 2016 14:49
Include the log file please
Denis Moreno
@Galithil
Nov 14 2016 14:49
@ewels
Phil Ewels
@ewels
Nov 14 2016 14:51
truncated by gitter, 2 secs
Félix C. Morency
@fmorency
Nov 14 2016 14:52
Pipeline completed at: Sat Nov 12 19:59:14 EST 2016
Execution status: OK
Execution duration: 1d 6h 36m 23s
\o/
Paolo Di Tommaso
@pditommaso
Nov 14 2016 15:02
could you re-execute the same with
nextflow -trace nextflow run .. etc
and upload the log again ?
Phil Ewels
@ewels
Nov 14 2016 15:08
yup
Need to wait for the job to get through the queue and run..
Denis Moreno
@Galithil
Nov 14 2016 15:14
Phil Ewels
@ewels
Nov 14 2016 15:19
Log file attached ^^
Paolo Di Tommaso
@pditommaso
Nov 14 2016 15:31
My fault, I should read my own documentation ..
path, cannot be null, you will need to use saveAs
publishDir path: "${params.outdir}/reference_genome", saveAs: { params.saveReference ? it : null }, mode: 'copy'
@Galithil have u tried to rename docker.config file to a different name eg docker0.config ?
Denis Moreno
@Galithil
Nov 14 2016 15:34
we're having the same issue, @ewels and me
Paolo Di Tommaso
@pditommaso
Nov 14 2016 15:35
please isolate the problem to a test case I can replicate and open an issue on GH
Denis Moreno
@Galithil
Nov 14 2016 15:37
it's probably the null thing
let's try to change it
Paolo Di Tommaso
@pditommaso
Nov 14 2016 15:37
if so try the latest snapshot I've uploaded
Denis Moreno
@Galithil
Nov 14 2016 15:37
yep
Paolo Di Tommaso
@pditommaso
Nov 14 2016 15:38
NXF_VER=0.22.5-SNAPSHOT nextflow run ... etc
Denis Moreno
@Galithil
Nov 14 2016 15:44
mmh it seems to have issues with some of our variables
i'll let @ewels look into that
thanks for the help
Paolo Di Tommaso
@pditommaso
Nov 14 2016 15:45
welcome
Phil Ewels
@ewels
Nov 14 2016 17:08
ok, so pulling the newest version gave me a whole load of warnings about unset param variables.. Got all of those fixed up and it's running now. So far so good.. fingers crossed!
I suspect that all of these undefined params were the root cause of a lot of our recent problems though.
Paolo Di Tommaso
@pditommaso
Nov 14 2016 17:09
ah, assign params to a default before use them!
:+1:
Phil Ewels
@ewels
Nov 14 2016 17:13
Yup, now done! Also replaced the cpus { params.makeHISATindex_cpus ?: 10 } stuff that we were talking about the other day with a simpler cpus { params.makeHISATindex_cpus } + earlier declaration with a default value of 10.
Not sure why I didn't do that in the first place tbh
Thanks for your help and patience :)
Paolo Di Tommaso
@pditommaso
Nov 14 2016 17:13
great
;)
Félix C. Morency
@fmorency
Nov 14 2016 18:54
@pditommaso if I set executor.queueSize=1, nextflow can't run tasks that have cpus >1
isn't that a bug?
Paolo Di Tommaso
@pditommaso
Nov 14 2016 20:33
@fmorency eventually, though I don't see any useful use case for having executor.queueSize=1
please open an issue for that on GH
Félix C. Morency
@fmorency
Nov 14 2016 20:33
run one task at the time
Paolo Di Tommaso
@pditommaso
Nov 14 2016 20:34
why to use a parallelisation framework to run one task at time? :)
Félix C. Morency
@fmorency
Nov 14 2016 20:35
debugging :)
Paolo Di Tommaso
@pditommaso
Nov 14 2016 20:35
I see, process.maxForks = 1 should do the same