These are chat archives for nextflow-io/nextflow

11th
Nov 2016
Phil Ewels
@ewels
Nov 11 2016 08:15
Hi @pditommaso, I think I've come across some buggy behaviour in nextflow (also tested by @viklund and @MaxUlysse)
$ ls
minimal.nf  nextflow.config

$ cat minimal.nf
process downloadFASTA {
    input: val dl_url from params.download_fasta
    when: params.download_fasta
    script:
    """
    curl -O -L $dl_url
    """
}

$ cat nextflow.config
process {
  stupid_var = params.p
}

$ nextflow run minimal.nf
N E X T F L O W  ~  version 0.22.4
Launching `minimal.nf` [elated_leakey] - revision: d08a065686
Nope
[warm up] executor > local
ERROR ~ No such variable: dl_url

 -- Check script 'minimal.nf' at line: 6 or see '.nextflow.log' file for more details
Johan Viklund
@viklund
Nov 11 2016 08:23
I have a smaller minimal.nf:
params.value = "hola"
process buggy {
    input:
        val v from params.value

    script:
    """
    echo $v
    """
}
Paolo Di Tommaso
@pditommaso
Nov 11 2016 08:29
minimal.nf is working fine to me, what are you expecting ?
Johan Viklund
@viklund
Nov 11 2016 08:30
heh, this rabbit hole is deeep
Paolo Di Tommaso
@pditommaso
Nov 11 2016 08:30
!
Johan Viklund
@viklund
Nov 11 2016 08:30
If you set params.value to false it should start failing
(even if you set it back to "hola" afterwards)
Paolo Di Tommaso
@pditommaso
Nov 11 2016 08:32
even if you set it back to "hola" afterwards
what do you mean? re-assign params.value to a different value?
Johan Viklund
@viklund
Nov 11 2016 08:33
no, 1. run it with false, look at it fail. 2. edit the source, set it to "hola" - watch it fail
"3". set params.p = "" in nextflow.config, watch it run
"4." remove params.p in nextflow.config, still runs
"5." set params.value to false - fails
Phil Ewels
@ewels
Nov 11 2016 08:34
1 and 5 work for me if I add the when statement I think
Johan Viklund
@viklund
Nov 11 2016 08:35
I think that was the process I went through at least
Phil Ewels
@ewels
Nov 11 2016 08:35
But if I set it to null or don't set it, I get the error
Paolo Di Tommaso
@pditommaso
Nov 11 2016 08:35
not able to replicate
please open an issue with a test case that I can replicate
tx
Johan Viklund
@viklund
Nov 11 2016 08:36
I'm working on it
@pditommaso did you run the buggy example or the minimal.nf? And did you use the config file?
(just want to make sure where you started so I can find the difference from my machine)
Johan Viklund
@viklund
Nov 11 2016 08:45
@pditommaso did you supply value on the command line?
I think the issue is with an empty params
(right now)
Phil Ewels
@ewels
Nov 11 2016 08:47
I made #242 with some steps I've reproduced myself a few times now
Paolo Di Tommaso
@pditommaso
Nov 11 2016 08:48
ok
Johan Viklund
@viklund
Nov 11 2016 09:05
I've posted some stuff in the issue as well
Paolo Di Tommaso
@pditommaso
Nov 11 2016 09:07
:+1:
Denis Moreno
@Galithil
Nov 11 2016 09:31
Hello, it looks like the nextflow.config in the main.nf dir is loaded over the one in my $CWD. Is this expected ?
correction : I have no idea which nextflow.config is being loaded
oh well.
Evan Floden
@evanfloden
Nov 11 2016 09:35
I think you can force a config with -c <config_fie>
If it helps with testing
Denis Moreno
@Galithil
Nov 11 2016 09:35
I'm already using that to add extra config
Denis Moreno
@Galithil
Nov 11 2016 10:20
It's interesting, the log tells me which config files are being used, and the actual config doesnt match the config files
does nextflow cache configs ?
Tiffany Delhomme
@tdelhomme
Nov 11 2016 10:22

this

.ifEmpty { error "Cannot find any bam file in: ${params.tumor_bam_folder}" }

does not stop the script if empty, is it the expected behaviour?

Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:26
does not stop the script if empty, is it the expected behaviour?
no, it should stop
can you please an issue for with a test case to replicate the problem?
does nextflow cache configs ?
no
Denis Moreno
@Galithil
Nov 11 2016 10:27
then dark forces are at work
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:28
definitely !
Tiffany Delhomme
@tdelhomme
Nov 11 2016 10:28
sure I will open it
Evan Floden
@evanfloden
Nov 11 2016 10:29
@tdelhomme: At a first guess, maybe the tumor_bam_folderexists but is empty in which case it may not stop
Tiffany Delhomme
@tdelhomme
Nov 11 2016 10:30
@pditommaso yees this is my case, I guessed error would stop in every case... so, how to stop it if exist but empty?
Denis Moreno
@Galithil
Nov 11 2016 10:32
am I allowed to have dashes in the parameters name in the configuration file ?
I guess not
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:32
nope
Denis Moreno
@Galithil
Nov 11 2016 10:32
that was the issue then
nextflow only reads what's on the right of the dash
how about underscores ?
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:33
but camel case identifiers are equivalent
underscore is fine
Denis Moreno
@Galithil
Nov 11 2016 10:33
I'm an underscore person. @ewels likes dashes
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:33
I mean, --hello-world is the same as params.helloWorld
Denis Moreno
@Galithil
Nov 11 2016 10:34
aha
Evan Floden
@evanfloden
Nov 11 2016 10:34
@tdelhomme: I think empty refers to the channel, so if the channel is empty it stops. In this sense, if the channel contains a directory that exists, NF is not aware of the contents. @pditommaso, best solution?
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:34
I know just like spaces vs tabs .. :)
Denis Moreno
@Galithil
Nov 11 2016 10:34
I was using profile{ngi-docker} and it kept suggesting me docker
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:34
ah wait
take in consideration, NF config file are compiled to groovy code
thus it follows BF/groovy syntax
thus the dashes are not allowed in identifiers
Denis Moreno
@Galithil
Nov 11 2016 10:36
I see
Evan Floden
@evanfloden
Nov 11 2016 10:36
@tdelhomme, you can add a closue to ifEmpty() though. So you could add a closure that trys to emit the contents of the directory. I find an example now.
Tiffany Delhomme
@tdelhomme
Nov 11 2016 10:37

@skptic this is my whole channel

tumor_bams = Channel.fromPath( params.tumor_bam_folder+'/*'+params.suffix_tumor+'.bam' )
              .ifEmpty { error "Cannot find any bam file in: ${params.tumor_bam_folder}" }

and this is not stopped if folder exists but is empty

Evan Floden
@evanfloden
Nov 11 2016 10:37
great
Denis Moreno
@Galithil
Nov 11 2016 10:37
I thought we were having a namespace conflict between our dockerprofile and the nextflow's dockerparam, but no hit. I still get
No such property: enabled for class: docker

groovy.lang.MissingPropertyException: No such property: enabled for class: docker
Tiffany Delhomme
@tdelhomme
Nov 11 2016 10:37
@skptic but i want this to be stopped if empty!
@skptic this makes the script not able to continue and also could not be stopped, a infinite time to complete the task
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:40
I think it's a bug
Denis Moreno
@Galithil
Nov 11 2016 10:42
I think I'm just doing something completely wrong
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:42
likely :)
could you post your config ?
Denis Moreno
@Galithil
Nov 11 2016 10:45
I have three, the one in my cwd, the one in the script dir and the one I provide on top
which one do you want ?
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:45
umm .. complex thing
Denis Moreno
@Galithil
Nov 11 2016 10:45
also, it seems to work if I move the nextflow script somewhere else so I only use two configs
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:46
you can use nextflow config to check the resulting configuration
Denis Moreno
@Galithil
Nov 11 2016 10:46
and by "it works" I mean "it breaks differently"
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:46
put them in a gist so I can give a look
Denis Moreno
@Galithil
Nov 11 2016 10:48
If you can wait a bit, I think I might have a clearer image of wha't going on
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:48
no hurry ..
Denis Moreno
@Galithil
Nov 11 2016 10:49
new error : Not a valid 'memory' value in process definition: 6gb nice
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:50
no b
Evan Floden
@evanfloden
Nov 11 2016 10:50
Here is a potential solution, not 100% sure it works and a bit ugly though.
tumor_bams = Channel.fromPath( params.tumor_bam_folder+'/*'+params.suffix_tumor+'.bam' )
              .map { it -> file(it + '*.bam') }
              .ifEmpty { error "Cannot find any bam file in: ${params.tumor_bam_folder}" }
              .map { it -> it.getParent() }
Denis Moreno
@Galithil
Nov 11 2016 10:50
I think the fact that nextflow tries to load the same config multiple times is what messes things up
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:51
what do u mean tries to load the same config multiple times ..
Denis Moreno
@Galithil
Nov 11 2016 10:53
I'll show you in a bit
Paolo Di Tommaso
@pditommaso
Nov 11 2016 10:53
need to leave now
put all your stuff in a gist or open an issue
Denis Moreno
@Galithil
Nov 11 2016 10:54
okay, see you
Evan Floden
@evanfloden
Nov 11 2016 10:55
@tdelhomme Any luck?
Tiffany Delhomme
@tdelhomme
Nov 11 2016 11:00
@skptic I have to go but will test in the afternoon and let you know, thanks a lot :smile:
Tiffany Delhomme
@tdelhomme
Nov 11 2016 12:29
@skptic I could also combine error with such a ugly thing like assert 0 == 1 to be sure I exit but...
Paolo Di Tommaso
@pditommaso
Nov 11 2016 14:46
@tdelhomme Use exit instead of `error``
Phil Ewels
@ewels
Nov 11 2016 15:25
@pditommaso - is there a way to supply a unit of memory on the command line with a parameter? I'm trying if( task.memory > params.m ){but getting java.lang.String cannot be cast to nextflow.util.MemoryUnit
Phil Ewels
@ewels
Nov 11 2016 15:32
nvm, got it: if( task.memory.toGiga() > params.m) {
Félix C. Morency
@fmorency
Nov 11 2016 17:13
just launched some processing using built-in ignite support. it works \o/
cool thing is if you have an active venv when launching the nextflow node, it takes that python version
Paolo Di Tommaso
@pditommaso
Nov 11 2016 18:19
:+1:
finally a good news today :smile:
Jason Byars
@jbyars
Nov 11 2016 22:16
What are the rules for the amazon cloud autoscale up? I.E. x jobs in queue for y minutes triggers adding workers?
Paolo Di Tommaso
@pditommaso
Nov 11 2016 22:18
On the road, check the last blog post
Jason Byars
@jbyars
Nov 11 2016 22:23
I see it. Thanks!