These are chat archives for nextflow-io/nextflow

21st
Sep 2016
Mike Smoot
@mes5k
Sep 21 2016 15:48
@pditommaso Hi Paolo, congrats on 0.22.0! I look forward to giving it a good workout shortly. In the meantime I'm still having a problem with this issue nextflow-io/nextflow#222. I've created a small test case that demonstrates my problem:
ch = Channel.from(0..10000)


process makeFile {

    input:
    val(num) from ch

    output:
    file("${num}.txt") into files

    script:
    """
    echo ${num} > ${num}.txt
    """
}

process passThrough {
    container 'ubuntu:16.04'

    input:
    file(f) from files

    output:
    file(f) into files_again

    script:
    """
    cat ${f}
    """
}

process finalStep {
    container 'ubuntu:16.04'

    input:
    file(f) from files_again

    output:
    stdout into cat_again

    script:
    """
    cat ${f}
    """
}
This will eventually fail like my actual pipeline. My question is, is the way that I forward a value from input to output in passThrough a legit thing to do? We do this in several places and I think its both easy to reason about and cleans up the DAG.
Paolo Di Tommaso
@pditommaso
Sep 21 2016 17:20
I'm trying to arrange a patch
Mike Smoot
@mes5k
Sep 21 2016 17:45
readlink -f would help if only BSD were more modern
Paolo Di Tommaso
@pditommaso
Sep 21 2016 17:46
God save Java
oops borken link
Paolo Di Tommaso
@pditommaso
Sep 21 2016 17:57
Ok, i've uploaded version 0.22.1-SNAPSHOT
Mike Smoot
@mes5k
Sep 21 2016 17:57
Awesome, will try!
Paolo Di Tommaso
@pditommaso
Sep 21 2016 17:58
bare in mind this uses the new caching mechanism, so resume won't work with old runs
Mike Smoot
@mes5k
Sep 21 2016 17:59
That's fine. I'll experiment first with my test case.
Paolo Di Tommaso
@pditommaso
Sep 21 2016 17:59
ok
Jason Byars
@jbyars
Sep 21 2016 22:02

What's the recommended way to deal with a process that takes an optional file? I.E. a variant caller that sometimes I want to use a hotspots file with? When the file isn't present, I don't want to error out and I really like having the reference file linked in the folder when the file is present. Something along these lines:

params.hotspots =""

HOTSPOTS=file(HOTSPOTS)

process dostuff {
  input:
  file HOTSPOTS

  output:
  file whatever

  script:
  """
  if [ -e $HOTSPOTS ]; then
    doit="--hotspots $HOTSPOTS"
  fi
  variantcaller --params blah blah blah $doit
  """
}

Sorry I'm sure I missed the relevant example.

Paolo Di Tommaso
@pditommaso
Sep 21 2016 22:06
yes, something like that
instead of a BASH if I would out at NF level
script:
def doit = params.hotspots ? "--hotspots $HOTSPOTS" : ''
"""
  variantcaller --params blah blah blah $doit
"""
Jason Byars
@jbyars
Sep 21 2016 22:23
that's exactly the idea. But how do I handle getting the file linked into the working directory? if params.hotspots isn't set and I do a file(params.hotspots) that should fail right?
Paolo Di Tommaso
@pditommaso
Sep 21 2016 22:24
file does not require the file to exist
not sure it would work with an empty string, but you can use a marker file name e.g. NONE
Jason Byars
@jbyars
Sep 21 2016 22:25
ahh, that should work. Let me test.
Mike Smoot
@mes5k
Sep 21 2016 22:49
@pditommaso Hi Paolo, I just finished testing your fix for nextflow-io/nextflow#222 and it looks great. I no longer see the errors I was seeing and I see the symlinks being fully resolved!