These are chat archives for nextflow-io/nextflow

7th
Mar 2017
Paolo Di Tommaso
@pditommaso
Mar 07 2017 17:48
@heuermh Hi Michael, we are interested as well!
what do you mean for "long-running services in CWL" ?
ahhh, you mean to launch the Spark daemon I guess ..
Félix C. Morency
@fmorency
Mar 07 2017 20:19
how does one skip a process if a channel is empty?
not sure how to use when: when channel is empty
Félix C. Morency
@fmorency
Mar 07 2017 20:45
nvm
Paolo Di Tommaso
@pditommaso
Mar 07 2017 21:01
An empty channel doesn't trigger any process execution
amacbride
@amacbride
Mar 07 2017 21:51
Is there any reason why a file output into two output channels instead of one would cause a script compilation error?
#!/usr/bin/env nextflow

process createFile {

        output:
                file ("downsample-output") into out1, out2

        """
        echo "5:10" > downsample-output
        """
}
Paolo Di Tommaso
@pditommaso
Mar 07 2017 21:52
that's a nasty issue
amacbride
@amacbride
Mar 07 2017 21:52
It seems to work fine when I only declare out1, but dies with the following error when I use it with 2 channels:
Paolo Di Tommaso
@pditommaso
Mar 07 2017 21:52
in a nutshell use
        output:
                file ("downsample-output") into (out1, out2)
nextflow-io/nextflow#141
amacbride
@amacbride
Mar 07 2017 21:52
Mar-07 13:50:03.033 [main] DEBUG nextflow.Session - Session aborted -- Cause: No such property: process for class: _nf_script_8af3d339
Mar-07 13:50:03.181 [main] ERROR nextflow.cli.Launcher - @unknown
groovy.lang.MissingPropertyException: No such property: process for class: _nf_script_8af3d339
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:53)
    at org.codehaus.groovy.runtime.callsite.PogoGetPropertySite.getProperty(PogoGetPropertySite.java:52)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:307)
    at _nf_script_8af3d339.run(_nf_script_8af3d339:3)
    at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:322)
    at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:156)
    at nextflow.cli.CmdRun.run(CmdRun.groovy:210)
    at nextflow.cli.Launcher.run(Launcher.groovy:410)
    at nextflow.cli.Launcher.main(Launcher.groovy:558)
Ah, gotcha. Thanks.
amacbride
@amacbride
Mar 07 2017 22:02

On a related note (since I'm reworking a stdin/stdout based process into a file-based one): is it possible to set a bash variable from within the script for later re-use?

I'm parsing a file received as input, then want to set that value as a variable for use within the same script.

essentially:
process read1 {
        echo true
        input:
                file ("downsample-output") from out1

        script:
        """
        echo "read1"
        someval=`cat downsample-output  | cut -f1 -d:`
        echo ${someval}
        """
}
Paolo Di Tommaso
@pditommaso
Mar 07 2017 22:06
Unfortunately no, you need to save it to a file
amacbride
@amacbride
Mar 07 2017 22:08
So how do I get the contents of a file into a variable?
file.readLines()?
Paolo Di Tommaso
@pditommaso
Mar 07 2017 22:09
You need to post-process later
for example
out1.map { it.text.trim() }.set { newChannel }
amacbride
@amacbride
Mar 07 2017 22:17
(looking puzzled) Is that an output specification?
Paolo Di Tommaso
@pditommaso
Mar 07 2017 22:19
nope, out1 is the channel in your previous example
you can transform the content from a file to a string value with that snippet
amacbride
@amacbride
Mar 07 2017 22:20
oh, so "from out1.map.blah blah" sets it into newChannel?
Paolo Di Tommaso
@pditommaso
Mar 07 2017 22:21
um
either
process foo {
  output: 'val_file' into out1

 """
 :
 echo val > val_file
}

out1.map { it.text.trim() }.set { newChannel }

process bar {
  input: val x from newChannel

  : etc
}
or
proces foo {
  output: 'val_file' into out1

 """
 :
 echo val > val_file
 """ 
}

process bar {
  input: val x from out1.map { it.text.trim() }

  : etc
}
amacbride
@amacbride
Mar 07 2017 22:34

Ah, got it. (As background, I wanted to replace my earlier stdin solution with a file-based one so it could be cached.)

This works:

process createFile {
    storeDir { "/tmp" }
    output:
        file ("downsample-output") into (out1, out2)

    """ 
    echo "5:10" > downsample-output
    """
}

process read1 {
    echo true
    input:
        val someval from out1.map { it.text.trim().tokenize(":")[0] }

    script:
    """ 
    echo "read1"
    echo $someval
    """
}

process read2 {
    echo true
    input:
        val someval from out2.map { it.text.trim().tokenize(":")[1] }

    script:
    """ 
    echo "read2"
    echo $someval
    """
}
Thanks!
Paolo Di Tommaso
@pditommaso
Mar 07 2017 22:35
:+1:
actually it should work also with stdout
in that case you won't need .text.trim()
amacbride
@amacbride
Mar 07 2017 22:36
I had been getting warnings like this:
StoreDir can only be used when using 'file' outputs
Paolo Di Tommaso
@pditommaso
Mar 07 2017 22:36
ah
yes
amacbride
@amacbride
Mar 07 2017 22:37
And since this happens to be a process that's time-consuming, but only needs to be done once per sample, I wanted to make sure I could cache it if need be.
Michael L Heuer
@heuermh
Mar 07 2017 22:39

ahhh, you mean to launch the Spark daemon I guess ..

Right, and possibly also the HDFS/Tachyon/Alluxio daemons.

Paolo Di Tommaso
@pditommaso
Mar 07 2017 22:40
uhh
I have no idea about CWL status about that
have you used Tachyon ?
Michael L Heuer
@heuermh
Mar 07 2017 22:41
we discussed it at the last BOSC conference but didn't commit to any action items, so I haven't thought about it since then
we've been moving towards just doing everything on Spark, with a new pipe API for streaming partitioned data in native formats to/from external apps. for us CWL would be more about reproducibility and integration with other pipelines.
Paolo Di Tommaso
@pditommaso
Mar 07 2017 22:44
interesting, is there any presentation/slides/something ?
Michael L Heuer
@heuermh
Mar 07 2017 22:49
ADAM presentation which mentions Spark-only variant calling pipeline https://t.co/UmRl2DR0BK
docs (in progress) on ADAM pipe API https://github.com/fnothaft/adam/blob/8eddeb0f8f035ca46e417536e47ae3ad444e14b1/docs/source/60_building_apps.md#using-adams-pipe-api-pipes
Paolo Di Tommaso
@pditommaso
Mar 07 2017 22:53
thanks