These are chat archives for nextflow-io/nextflow

10th
May 2019
Rad Suchecki
@rsuchecki
May 10 00:26
Also curious about your answer to @micans question @csawye01
Channels created explicitly (using .create()) need to be explicitly closed (with .close()) otherwise you end up with an indefinite wait. I didn't think you can put anything into a channel created with Channel.empty()? My understanding of its application is limited to http://nextflow-io.github.io/patterns/index.html#_skip_process_execution
Adrian Viehweger
@phiweger
May 10 10:55
is it possible to specify the nextflow config using json format? I would like to create config templates which allow dynamic changes e.g. of which container to use.
Adrian Viehweger
@phiweger
May 10 11:21
I guess I am asking "is it possible to use placeholders in the config file, and if so, where can I find some documentation on this"? thank you
Paolo Di Tommaso
@pditommaso
May 10 11:46
not the complete config file, but script parameters can be provides as a json or yaml file using the -params-file option
check nextflow run -h
Chelsea Sawyer
@csawye01
May 10 13:13
@micans I'm funnelling several channels that all may or may not exist into one channel for a process to use. It's tricky as all of the channels have the potential to not exist so this seemed like the best option.
@rsuchecki It works when I mix things into the empty channel. I'm not sure if its better to use Channel.create() and .close() or just mix everything into the empty channel.
empty_ch = Channel.empty()
All_ch = empty_ch.mix(first_ch, second_ch)
micans
@micans
May 10 13:16
@csawye01 can you not use All_ch = first_ch.mix(second_ch)?
Chelsea Sawyer
@csawye01
May 10 13:17
@micans No as the first_ch may not have been created.
micans
@micans
May 10 13:18
But then I'd expect that not to work in your code either, does it!?
Chelsea Sawyer
@csawye01
May 10 13:19
@micans mixing into the empty channel works surprisingly
micans
@micans
May 10 13:19
Cool!
Chelsea Sawyer
@csawye01
May 10 13:21
@micans haha it was! I was a bit stuck on how to almost all processes that may run or may not depending on the input in the beginning of the pipeline and how to converge them into a process further down. So very happy when this worked!
Stephen Kelly
@stevekm
May 10 17:17
@pditommaso @evanfloden what is up with the new changes to the latest version of Nextflow? It has some green system resource metrics in the terminal now, which is cool. But it seems to be trying to run cached versions of the main.nf script. I was trying to demo Nextflow to someone yesterday and I would update the script and re-run it (nextflow run main.nf) but the changes to the script were not happening, it kept running old versions of the script. At one point, I made a change to the script and ran nextflow run main.nf three times consecutively, and each time it ran a different version of the script, all but the last one being different from what was actually saved to disk
I think it was version 19.04. I use 19.01 for all my production stuff.
Paolo Di Tommaso
@pditommaso
May 10 17:49
if don't like disable it, if it's a bug open an issue, if you have a suggestion to improve it happy to listen it
Tobias "Tobi" Schraink
@tobsecret
May 10 18:11
How does one specify the following :
conda install -c bioconda/label/cf201901 bioconductor-signer in the conda directive in nextflow?
bioconda/label/cfc201901::bioconductor-signer does not seem to work
Tobias "Tobi" Schraink
@tobsecret
May 10 18:20
Ugh figured out it's that the version of the package isn't available for some reason, my bad
Stephen Kelly
@stevekm
May 10 18:20
@pditommaso the ANSI logging is fine, what I was referring to was that Nextflow does not appear to be running the actual Nextflow script anymore, but instead must be using some cached copy, because changes to the script are not being reflected in what is actually getting run. For example, when I change the contents of .subscribe { println "[foo] ${it}" }, I have to run nextflow run main.nf at least twice before it actually starts showing that in the output. The same for adding or removing process's
Paolo Di Tommaso
@pditommaso
May 10 18:22
version + log file + test case
Stephen Kelly
@stevekm
May 10 18:22
@tobsecret make sure you check what channels you have available in conda, when I first migrated from phoenix to Big Purple I found there were actually some combinations of packages that were no longer available, for some reason certain channels were not present
Tobias "Tobi" Schraink
@tobsecret
May 10 18:23
Yeah, no I figured it out - I was asking for the wrong version of the package - had nothing to do with conda
gonna try it with biocontainers now
Stephen Kelly
@stevekm
May 10 18:24
@pditommaso I will try to get some test cases and logs, though I was also wondering what sort of feature might have been added that caused such a thing? I was looking through the Nextflow updates history and could not find mention of any new features that sounded relevant to this
it worked fine in 19.01 so it must be recent
Tobias "Tobi" Schraink
@tobsecret
May 10 18:42
@stevekm ever had the problem of NextFlow not finding files from bin that are executables while using singularity containers? Tried autoMounts = true, does not solve the problem
iirc, NF automatically converts docker containers to singularity, right?!
or is that just nf-core magic?
Tobias "Tobi" Schraink
@tobsecret
May 10 18:47
Huh, super weird!!! only works with absolute path to the nextflow executable
Stephen Kelly
@stevekm
May 10 18:47
I am not aware of any way to convert a docker container to singularity "automatically"
the method I use requires many steps and root access; https://github.com/stevekm/docker2singularity-demo
its easier just to have separate singularity and docker recipes and build them independently
Tobias "Tobi" Schraink
@tobsecret
May 10 18:49
Hmmm, no it seems to just work
Stephen Kelly
@stevekm
May 10 18:50
no clue what its doing then
Tobias "Tobi" Schraink
@tobsecret
May 10 18:50
me neither
Olga Botvinnik
@olgabot
May 10 18:51
Hello, I am having trouble with memory specification. Here is the command and error:
 ✘  Fri 10 May - 11:49  ~/code/kmer-hashing/kh-workflows/nf-kmer-similarity/tabula-muris   origin ☊ olgabot/index-maca 52‒ 
  nextflow run czbiohub/nf-kmer-similarity -latest -profile aws \
        --work-dir s3://kmer-hashing/nextflow-intermediates/ \
        --samples samples.csv \
        --outdir s3://kmer-hashing/tabula_muris/ \
        -r bf5b76dd7f61909fca3dc0c29b2207503e7cc61c \
        --create_sbt_index --log2_sketch_sizes 12 \
        --molecules protein --ksizes 21 -resume \
        --no_compare --e.process.memory 16.GB
N E X T F L O W  ~  version 19.03.0-edge
ERROR ~ Project config file is malformed -- Cause: No signature of method: nextflow.util.MemoryUnit.multiply() is applicable for argument types: (ConfigObject) values: [[:]]
Possible solutions: multiply(java.lang.Number)
the config file is:
docker.enabled = true

timeline {
  enabled = true
}
report {
  enabled = true
}
trace {
  enabled = true
}
dag {
  enabled = true
}



profiles {

  local {
    process.executor = 'local'
  }

  docker {
    process.executor = 'local'
    docker.enabled = true
  }

  aws {
    process.executor = 'awsbatch'
    process.queue = 'nextflow'
    executor.awscli = '/home/ec2-user/miniconda/bin/aws'
    workDir = 's3://czb-nextflow/work'
    docker.enabled = true
    params.tracedir = './'
  }
  test {
  params {
    // Input data
    samples = 'testing/samples.csv'
    fastas = 'testing/fastas/*.fasta'
    ksizes = '3,9'
    log2_sketch_sizes = '2,4'
    molecules = 'dna,protein'
    read_pairs = 'testing/fastqs/*{1,2}.fastq.gz'
    sra = "SRP016501"
    create_sbt_index = True
    }
  }
}

// Capture exit codes from upstream processes when piping
process.shell = ['/bin/bash', '-euo', 'pipefail']


aws {
    region = 'us-west-2'

    client {
        maxConnections = 20
        connectionTimeout = 10000
        uploadStorageClass = 'INTELLIGENT_TIERING'
        storageEncryption = 'AES256'
    }
}


process {


  maxRetries = 3
  maxErrors = '-1'

  // Resource requirements
  withName: sourmash_compare_sketches {
    memory = 512.GB
    cpus = 1
  }
  withName: sourmash_compute_sketch {
    memory = 4.GB
    cpus = 1
  }
  // Resource requirements
  withName: sourmash_index_sketches {
    memory = 64.GB
    cpus = 1
  }
}
I can't understand what is being multiplied by what. I used to have the check_max function in there to multiply by task.attempt but it's not there anymore so I don't understand the command at all
Stephen Kelly
@stevekm
May 10 18:52

@tobsecret

Huh, super weird!!! only works with absolute path to the nextflow executable

what only works? If your bin dir is not accessible in the Singularity container then it means that your $PATH is not propagating correctly

Olga Botvinnik
@olgabot
May 10 18:53
What is even more confusing is that this workflow has no errors locally but cannot run on the remote machine
And I've renamed all nextflow.config files to nextflow.config.backup so local ones shouldn't get loaded
please help!
this worked fine a few days ago and I can't figure out what went wrong
Tobias "Tobi" Schraink
@tobsecret
May 10 18:55

@stevekm
sorry, should have explained:
when running

nextflow run <symlink to projectDir>/main.nf -profile prince -resume, NF does not find the files in the bin directory

When running instead:

nextflow run <absolute path to projectDir>/main.nf -profile prince -resume, NF has no problem finding it

Olga Botvinnik
@olgabot
May 10 18:56
Is there a way to find out exactly which line of the config is failing?
Tobias "Tobi" Schraink
@tobsecret
May 10 18:56
@olgabot yeah, no clue
Looks like it's finding some other config file as well
why is maxErrors = '-1' though?
That seems weird
Olga Botvinnik
@olgabot
May 10 18:58
idk I copied that
what does that mean?
okay looking for other files
 Fri 10 May - 11:58~/code/kmer-hashing/kh-workflows/nf-kmer-similarity/tabula-muris   origin ☊ olgabot/index-maca 52‒ 
  find ~/ -name 'nextflow.config'                                                                                                          /home/olga/.local/share/Trash/files/kh-workflows/nf-kmer-similarity/nextflow.config
/home/olga/.nextflow/assets/pbelmann/sra-download/nextflow.config
/home/olga/.nextflow/assets/czbiohub/nf-kmer-similarity/nextflow.config
/home/olga/.nextflow/assets/czbiohub/sra-download/nextflow.config
/home/olga/.nextflow/assets/biocorecrg/indrop/nextflow.config
And for some reason /home/olga/.nextflow/assets/czbiohub/nf-kmer-similarity/nextflow.config isn't updated to the latest version
yeah wtf it wasn't pulled
Tobias "Tobi" Schraink
@tobsecret
May 10 19:00
Mystery solved? :sweat_smile:
Olga Botvinnik
@olgabot
May 10 19:00
wow okay I pulled the latest version and now it's fine
Tobias "Tobi" Schraink
@tobsecret
May 10 19:00
:raised_hands:
Laurence E. Bernstein
@lebernstein
May 10 19:01
Hi Olga <wave> :-)
Olga Botvinnik
@olgabot
May 10 19:02
wtf so the issue was that if the config file in the current revision is invalid, it fails
Hi @lebernstein !
okay I'll file a bug
Tobias "Tobi" Schraink
@tobsecret
May 10 19:04
I don't quite understand? so you had some old config file laying around and it used that?
Olga Botvinnik
@olgabot
May 10 19:04
it wasn't listening to -latest so the most recent config file was never pulled
it read the existing ~/.nextflow/assets/czbiohub/nf-kmer-similarity/nextflow.config and failed
but never pulled the latest revision
Tobias "Tobi" Schraink
@tobsecret
May 10 19:04
ooooooh
Olga Botvinnik
@olgabot
May 10 19:04
even when I manually specified the revision with a hash
Tobias "Tobi" Schraink
@tobsecret
May 10 19:04
gotcha, you were executing from a GH repo
Olga Botvinnik
@olgabot
May 10 19:05
yep
Tobias "Tobi" Schraink
@tobsecret
May 10 19:05
yeah, that's odd
def a bug
Olga Botvinnik
@olgabot
May 10 19:05
whelp there went one hour
Tobias "Tobi" Schraink
@tobsecret
May 10 19:06
:sweat_smile: at least it was only one hour
Olga Botvinnik
@olgabot
May 10 19:07
haha
Tobias "Tobi" Schraink
@tobsecret
May 10 19:08
I spent a day trying to figure out a way to install 1 (!) R-package and dependencies - ended up conceding and am using a docker container now :fire:
Olga Botvinnik
@olgabot
May 10 19:11
wowww
I hate computers
Stephen Kelly
@stevekm
May 10 19:17

@stevekm
sorry, should have explained:
when running

nextflow run <symlink to projectDir>/main.nf -profile prince -resume, NF does not find the files in the bin directory

When running instead:

nextflow run <absolute path to projectDir>/main.nf -profile prince -resume, NF has no problem finding it

@tobsecret I just put the nextflow executable in the directory where the project is... I think its better that way too because then you do not try to run the same workflow with two different versions of nextflow accidentally, seems like asking for trouble

I spent a day trying to figure out a way to install 1 (!) R-package and dependencies - ended up conceding and am using a docker container now

yea you def want to try conda inside Docker/Singularity in those situations, its not worth bothering to try getting it installed any other way esp. with R

Tobias "Tobi" Schraink
@tobsecret
May 10 19:23
@stevekm I am running things from scratch but keep the project dirs on home bc scratch is flushed on prince (supposedly) and I don't have enough space/ # of files allowed on home