These are chat archives for nextflow-io/nextflow

15th
Jan 2019
Robin
@rduque1
Jan 15 15:23
Hi, does someone knows if we can have ./nextflow dir also mounted on s3 as for the workdir. If I resume a job apparently this folder is needed too to be able to resume. The workdir isn't enough.
KochTobi
@KochTobi
Jan 15 15:38
you could try to mount your s3 bucket to your instance although I would suspect that the latency is to high. Why don't you just use the same instance to resume the pipeline?
Robin
@rduque1
Jan 15 15:50
@KochTobi ok thanks, it because the instance is terminated automatically by managing process.
Bede Constantinides
@bede
Jan 15 16:14
nf is running fine but spitting out Unable to find any JVMs matching version "1.8"after a fresh install on Mac OS 10.14 with JDK 11.0.1. May I ignore?
Paolo Di Tommaso
@pditommaso
Jan 15 17:22
weird, it should not happen, what NF version are you using ?
Stephen Kelly
@stevekm
Jan 15 19:26

is there a way to pass arbitrary variables into the process config? Like this:

nextflow.config

process {
    withName: task1 {
                foo = "bar"
            }
}

main.nf

process task1 {
output:
file("${output_file}")

script:
output_file = "${foo}"
"""
touch "${output_file}"
"""
}
I want to make a dictionary or something of the output filenames and output file suffixes to use for processes, but there is nothing I currently available in the process config that I can use, but it would work if maybe I could set some value in the nextflow.config and then use it in the main pipeline script
otherwise I will have to use an external JSON file, but that is dangerous because it will become out of date very quickly. Would be much easier if I could just bundle it into the nextflow.config
Stephen Kelly
@stevekm
Jan 15 20:17

so apparently this works:

nextflow.config

process {
    withName: task1 {
        echo = true
        foo = "bar"
            }
    withName: task2 {
        echo = true
        foo = "baz"
            }
}

main.nf

Channel.from(['a', 'b', 'c']).into { input_ch; input_ch2 }


process task1 {
    tag "${sampleID}"
    input:
    val(sampleID) from input_ch

    script:
    output_bam = "${sampleID}.bam"
    """
    echo "[task1] ${sampleID}: ${task.foo}"
    touch "${sampleID}.bam"
    """

}
process task2 {
    tag "${sampleID}"
    input:
    val(sampleID) from input_ch2

    script:
    output_bam = "${sampleID}.bam"
    """
    echo "[task2] ${sampleID}: ${task.foo}"
    touch "${sampleID}.bam"
    """
}

output

./nextflow run main.nf
N E X T F L O W  ~  version 18.10.1
Launching `main.nf` [infallible_yalow] - revision: 29e5ccad7f
WARN: Unknown directive `foo` for process `task1`
[warm up] executor > local
WARN: Unknown directive `foo` for process `task2`
[3c/76c2d7] Submitted process > task2 (b)
[87/6bd062] Submitted process > task1 (a)
[2e/a8e8a1] Submitted process > task1 (b)
[71/9e868e] Submitted process > task2 (c)
[a1/4fd00c] Submitted process > task2 (a)
[e8/6d40de] Submitted process > task1 (c)
[task1] b: bar
[task2] c: baz
[task2] a: baz
[task1] c: bar
[task2] b: baz
[task1] a: bar

Is this safe to do?

In my nextflow.config I would like to have filename suffixes saved for each process, then call them like this within the main pipeline script.

Tobias Neumann
@t-neumann
Jan 15 21:10
@tbugfinder Thanks - that was the trick. I used the name of the created key-pair in the AWS console and then the private key from the .pemfile to connect. One crucial thing that is also not mentioned in the documentation is to specify an appropriate securityGroup that allows connections from 0.0.0.0/0 because the default securityGroup will block any SSH access from the web
Tobias Neumann
@t-neumann
Jan 15 21:19
So after cluster creation with nextflow cloud create I have created two clusters which always prompt me the connection string after creation. With nextflow cloud list I can list all available clusters again, but where would I find the connection strings again?