These are chat archives for nextflow-io/nextflow

14th
May 2018
Vladimir Kiselev
@wikiselev
May 14 2018 10:06
Hi @pditommaso, does beforeScript directive accept running script files from bin folder? E.g. beforeScript = “sh myiinit.sh”. Does not work for us…
Caspar
@caspargross
May 14 2018 10:12
Hi! Sometimes i have the problem that Nextflow doesnt finish the pipeline correctly. All processes are finieshed but the workflow.onComplete event handler is not executed. Any one of you experienced the problem before?
Paolo Di Tommaso
@pditommaso
May 14 2018 11:53
Never seen. If you are able to replicate the error please open an issue with a test case.
Edgar
@edgano
May 14 2018 11:55
@caspargross Can you post the onComplete code? Sometime I have similar behaviour but it was sintax mistake on the code
Sven F.
@sven1103
May 14 2018 16:06
@pditommaso any good way for me to render the docs for testing?
make html raises:
sphinx-build -b html -d _build/doctrees   . _build/html
make: sphinx-build: Command not found
make: *** [Makefile:45: html] Error 127
ah, wait
it helps to install sphinx in the first place -.- sorry for the spam
Paolo Di Tommaso
@pditommaso
May 14 2018 16:35
yes exactly
Sven F.
@sven1103
May 14 2018 16:39
ok, done with some docs for #681
Paolo Di Tommaso
@pditommaso
May 14 2018 16:39
well done
Sven F.
@sven1103
May 14 2018 16:39
will check later for comments
thank you :)
Paolo Di Tommaso
@pditommaso
May 14 2018 16:40
no, I say thank you !
Sven F.
@sven1103
May 14 2018 16:40
this was fun :)
Paolo Di Tommaso
@pditommaso
May 14 2018 16:42
:smile:
Félix C. Morency
@fmorency
May 14 2018 18:43
What would be the cleanest way of checking if a channel is empty in a when: statement?
Paolo Di Tommaso
@pditommaso
May 14 2018 18:44
if a channel is empty, the task is not executed, therefore you don't need to check it
Félix C. Morency
@fmorency
May 14 2018 18:45
But I want to execute a task Foo if channel bar is empty
Paolo Di Tommaso
@pditommaso
May 14 2018 18:48
mmm, you need some trick like
process foo {
  input:
  val x from bar.ifEmpty { 'EMPTY'  } 
  when:
  x == 'EMTPY' 
  ... 
}
Félix C. Morency
@fmorency
May 14 2018 18:53
Mmmm I think this won't work because my use-case involves processing Foo multiple times and the channel bar will already be consumed?
Paolo Di Tommaso
@pditommaso
May 14 2018 18:58
this works
Channel.empty().set { bar }

process foo {
  input:
  val x from bar.ifEmpty { 'EMPTY'  } 
  when:
  x == 'EMPTY'

  script:
  '''
  echo hello
  ''' 
}
Félix C. Morency
@fmorency
May 14 2018 18:58
Gtg in a meeting, I will post an example after
Shawn Rynearson
@srynobio
May 14 2018 18:59

I wanted to check that my thought process is correct.

When you are using aws-batch and you define a param in your config file (example: a reference fasta), and you use the param in your process, the file will not automatically be transferred to the compute node. You have to create a input each time you want to use the file, so the command.run script can add it to the s3 download.

Paolo Di Tommaso
@pditommaso
May 14 2018 18:59
yep
only input: file X are downloaded
Shawn Rynearson
@srynobio
May 14 2018 19:00
@pditommaso thanks! Thought so, but I wanted to make sure I'm not adding a bunch of redundant inputs, if their was a better method.
Paolo Di Tommaso
@pditommaso
May 14 2018 19:01
:+1:
Vladimir Kiselev
@wikiselev
May 14 2018 20:01
hi @pditommaso, what this could be? May-14 17:20:18.387 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Process irods > Skipping output binding because one or more optional files are missing: fileoutparam<0:1>
my process is this:
process irods {
    tag "${sample}"

    input: 
        val sample from sample_list.flatMap{ it.readLines() }
    output: 
        set val(sample), file('*.cram') optional true into cram_files
    script:
    """
    irods.sh ${sample}
    """
}
where irods.sh is in the bin folder and executable
the process did not produce any errors
Paolo Di Tommaso
@pditommaso
May 14 2018 20:03
what are you expecting ?
Vladimir Kiselev
@wikiselev
May 14 2018 20:05
if I run irods.sh manually it pulls files from the irods database
So .command.sh looks ok:
#!/bin/bash -euo pipefail
irods.sh ImmHet6564352
Paolo Di Tommaso
@pditommaso
May 14 2018 20:06
that's supposed to get the *.cram files ?
Vladimir Kiselev
@wikiselev
May 14 2018 20:07
however, .command.err has this:
/lustre/scratch117/cellgen/cellgeni/tic-24test/work/bf/3928e3e5183d31f751e0063bb0aac9/.command.stub: line 99: 12058 Terminated              nxf_trace "$pid" .command.trace
yes
Paolo Di Tommaso
@pditommaso
May 14 2018 20:07
do they exist in the /lustre/scratch117/cellgen/cellgeni/tic-24test/work/bf/3928e3e5183d31f751e0063bb0aac9/ path ?
Vladimir Kiselev
@wikiselev
May 14 2018 20:08
Nope, they are missing and I can’t understand why
Paolo Di Tommaso
@pditommaso
May 14 2018 20:08
do the irods.sh is expected to copy in the current path ?
Vladimir Kiselev
@wikiselev
May 14 2018 20:09
yes
Paolo Di Tommaso
@pditommaso
May 14 2018 20:09
is this k8s execution ?
Vladimir Kiselev
@wikiselev
May 14 2018 20:10
No, sorry, this is on the farm, we just changed the way we call irods and moved commands to a file
Paolo Di Tommaso
@pditommaso
May 14 2018 20:10
LSF ?
Vladimir Kiselev
@wikiselev
May 14 2018 20:10
yes
Félix C. Morency
@fmorency
May 14 2018 20:10
Channel.from('a', 'b', 'c', 'd').set{foo}
Channel.empty().set{bar}

process Foo {
    input:
    val x from foo
    val e from bar.ifEmpty('EMPTY')

    output:
    file "DONE"

    when:
    e == 'EMPTY'

    ...
}
In this example, Foo will only be executed once.
Paolo Di Tommaso
@pditommaso
May 14 2018 20:11
@wikiselev change in that directory, and try to execute as a plain LSF job ie. bsub < .command.run
there should be something not working
Vladimir Kiselev
@wikiselev
May 14 2018 20:12
yep just did it and I think I found the problem, digging it now, thanks a lot!
Paolo Di Tommaso
@pditommaso
May 14 2018 20:13
that's the first thing to do ! :)
Vladimir Kiselev
@wikiselev
May 14 2018 20:13
I know, I am very bad, sorry, and you are too good and responsive!
Paolo Di Tommaso
@pditommaso
May 14 2018 20:13
:smile:
Vladimir Kiselev
@wikiselev
May 14 2018 20:14
I actually regularly stop myself from writing here, but today was too stresfull ;-)
Paolo Di Tommaso
@pditommaso
May 14 2018 20:14
ahah
Vladimir Kiselev
@wikiselev
May 14 2018 20:14
thanks for you huge help, anyway!
Paolo Di Tommaso
@pditommaso
May 14 2018 20:14
I know, better to share the pain !
you are welcome
Vladimir Kiselev
@wikiselev
May 14 2018 20:14
yep
Paolo Di Tommaso
@pditommaso
May 14 2018 20:18
@fmorency y is not defined
Félix C. Morency
@fmorency
May 14 2018 20:19
here I fixed the example
(this is just a toy example to show you the behaviour)
Paolo Di Tommaso
@pditommaso
May 14 2018 20:19
I guess if you use val e from bar.ifEmpty('EMPTY').first() it works as you are expecting
Félix C. Morency
@fmorency
May 14 2018 20:21
Yup it seems to work! Thanks!
Paolo Di Tommaso
@pditommaso
May 14 2018 20:21
:+1: