These are chat archives for nextflow-io/nextflow

14th
Jun 2017
Michael Milton
@TMiguelT
Jun 14 2017 06:47
Hi all, I notice the DNAnexus executor has been deprecated. Is this because it doesn't work properly, or was it just because of the maintenance burden?
Paolo Di Tommaso
@pditommaso
Jun 14 2017 07:02
Hi, because of the maintenance burden associated with the fact that we are not aware of any real user
Michael Milton
@TMiguelT
Jun 14 2017 07:04
Interesting. That's totally understandable
Paolo Di Tommaso
@pditommaso
Jun 14 2017 07:05
are you a DNAnexus user ?
Michael Milton
@TMiguelT
Jun 14 2017 07:05
We're more of a potential DNAnexus user
Paolo Di Tommaso
@pditommaso
Jun 14 2017 07:06
Are you a DNAnexus investor ? :)
Michael Milton
@TMiguelT
Jun 14 2017 07:06
Hmm?
Paolo Di Tommaso
@pditommaso
Jun 14 2017 07:07
kidding because I'm not understanding by We're more of a potential DNAnexus user
Michael Milton
@TMiguelT
Jun 14 2017 07:10
Oh just that my organization looks like it's moving to DNAnexus, so I'm looking into frameworks that can run on DNAnexus and also elsewhere
Paolo Di Tommaso
@pditommaso
Jun 14 2017 07:10
I see
Félix C. Morency
@fmorency
Jun 14 2017 13:51
@pditommaso quick question. We usually start NF from the folder we want the outputs to be. Is there a way to start NF elsewhere and specify the output folder from the cmd line?
Phil Ewels
@ewels
Jun 14 2017 14:02
eg. publishDir "${params.outdir}/fastqc", mode: 'copy'
@pditommaso - I have a groovy / NF function scoping question
eg. the following test workflow:
cheers=Channel.from "Bonjour", "Ciao","Hello","Hola"
process sayHello {
    input: val x from cheers
    script: "echo $x world!"
}

def foo(){ log.info "BAR" }
foo()

workflow.onComplete {
    log.info "FOO"
    foo()
}
[warm up] executor > local
BAR
[b8/27fa9e] Submitted process > sayHello (2)
[cf/990bde] Submitted process > sayHello (1)
[ef/5f2e8a] Submitted process > sayHello (3)
[01/23fd48] Submitted process > sayHello (4)
FOO
ERROR ~ Failed to invoke `workflow.onComplete` event handler
Félix C. Morency
@fmorency
Jun 14 2017 14:05
@ewels Not just the publishDir but also the workDir. Anything that NF outputs when ran from a folder
Phil Ewels
@ewels
Jun 14 2017 14:06
With the main error log:
Jun-14 16:04:47.375 [main] ERROR nextflow.script.WorkflowMetadata - Failed to invoke `workflow.onComplete` event handler
groovy.lang.MissingMethodException: No signature of method: nextflow.util.ReadOnlyMap.foo() is applicable for argument types: () values: []
Possible solutions: any(), find(), sort(), get(java.lang.Object), max(groovy.lang.Closure), find(groovy.lang.Closure)
@fmorency - you can set the workDir with workDir inside the pipeline, or -w on the command line, or NXF_WORK in the env vars
You'll still get .nextlow.log and .nextflow output in the cwd I think, not sure if it's possible to avoid that though
Maxime Garcia
@MaxUlysse
Jun 14 2017 14:10
@ewels I'm guessing you can't use Channels once the workflow is complete
Phil Ewels
@ewels
Jun 14 2017 14:11
The channel is just a holder to get the script to run - it's a function that I'm trying to call
Maxime Garcia
@MaxUlysse
Jun 14 2017 14:11
I'm guessing you can't use functions once the workflow is complete
I remember trying using a fonction in the workflow.onComplete and having the same problem
Phil Ewels
@ewels
Jun 14 2017 14:12
fair enough :+1:
Paolo Di Tommaso
@pditommaso
Jun 14 2017 14:12
I lost the party, could you summarise what's the problem ?
Maxime Garcia
@MaxUlysse
Jun 14 2017 14:13
@pditommaso Can we use a function in a workflow.onClomplete?
Paolo Di Tommaso
@pditommaso
Jun 14 2017 14:13
as main logic in place of a closure ?
Phil Ewels
@ewels
Jun 14 2017 14:14
I have a minimal example above ^^
Paolo Di Tommaso
@pditommaso
Jun 14 2017 14:14
it should work, let me check
it works if you specify this eg

workflow.onComplete {
    log.info "FOO"
    this.foo()
}
however it sounds like a bug
I will investigate
Maxime Garcia
@MaxUlysse
Jun 14 2017 14:17
Oo
Thanks a lot
Phil Ewels
@ewels
Jun 14 2017 14:17
:bug: :tada:
Paolo Di Tommaso
@pditommaso
Jun 14 2017 14:17
:)
the reason is this
but I don't remember any more if it's a bug or a feature :)
Maxime Garcia
@MaxUlysse
Jun 14 2017 14:19
I'm voting for a feature
Paolo Di Tommaso
@pditommaso
Jun 14 2017 14:20
feature in the meaning that should not access the external context by default ;)
Phil Ewels
@ewels
Jun 14 2017 14:26
hah, my vote is for a bug ;)
Maxime Garcia
@MaxUlysse
Jun 14 2017 14:27
I'm guessing it will depend if @pditommaso intends to correct it or not
Phil Ewels
@ewels
Jun 14 2017 14:27
I have terrible amnesia with previous coding work - once I made significant progress through a new MultiQC module before realising that I'd already written it :sweat:
Phil Ewels
@ewels
Jun 14 2017 15:16
Hi @pditommaso - I just finished putting together a minimal working example of a NF pipeline that can send proper multipart MIME e-mails (with embedded images) on pipeline completion: https://gist.github.com/ewels/2da5023f0b045f92434ad0cb5ad7e8aa
It uses pure groovy instead of any command line tools, so hopefully a bit more fancy
I was thinking that this could be the starting point for some kind of core functionality.. Do you want me to put it somewhere?
Paolo Di Tommaso
@pditommaso
Jun 14 2017 15:18
you are a groovy pro :)
Phil Ewels
@ewels
Jun 14 2017 15:18
hah, not sure about that..
there is a lot of Stack Overflow in the above :laughing:
(and even more trial and error)
Paolo Di Tommaso
@pditommaso
Jun 14 2017 15:19
from where you are getting smtp informations to send the email ?
Phil Ewels
@ewels
Jun 14 2017 15:19
setting it as defaults for localhost at the top of main.nf, but I had to set this manually to get it to work on my system (using a gmail smtp). Hopefully localhost works on linux systems..? Not sure.
Shellfishgene
@Shellfishgene
Jun 14 2017 15:20
Where can I set what nf reports to stdin? Can't find it in the docs...
I'd just like to get a line when a job finishes, not just submission
Paolo Di Tommaso
@pditommaso
Jun 14 2017 15:25
@ewels this is something I would like to avoid ..
then I would expect people that want to send Slack notifications, etc :)
tonywang0613
@tonywang0613
Jun 14 2017 15:26
Hi , I started to use the nextflow last week, it's awesome. I have a question, I have several process, each process generate several outputs, can I pipe those files into one channel, eg. every process have file(sample) into uploadfiles, then I have another process handle upload, eg. file(sample) from uploadfiles
Paolo Di Tommaso
@pditommaso
Jun 14 2017 15:27
@ewels what about improving the template generation and rely on the standard mail? Anyhow open an issue and will continue the discussion there
Shellfishgene
@Shellfishgene
Jun 14 2017 15:32
@ewels In your RNASeq workflow, do you use the mapping just for the quality checks, or later also to count reads?
Phil Ewels
@ewels
Jun 14 2017 15:34
@pditommaso - yes, I'm using mail in the RNA pipeline currently but it handles multipart (HTML) emails differently on different systems (Linux / mac). I thought pure groovy may be more consistent. Will make an issue anyway, but will be end if next week now as travelling
@Shellfishgene - mapping is required for all downstream processes, including (gene) read counts yes
Is that what you mean?
Shellfishgene
@Shellfishgene
Jun 14 2017 15:36
@ewels I meant if you actually use the STAR mapping against the genome for counting - I think for example RSEM wants the reads mapped against a transciptome (that it generates itself from the annotation). I was just wondering what you do.
Félix C. Morency
@fmorency
Jun 14 2017 15:39
We're using
workflow.onComplete {
    log.info "Pipeline completed at: $workflow.complete"
    log.info "Execution status: ${ workflow.success ? 'OK' : 'failed' }"
    log.info "Execution duration: $workflow.duration"

    if(params.mail_on_complete) {
        def subject = "Human pipeline execution (${ workflow.success ? 'SUCCESS' : 'FAILED' })"

        ['mailx', '-r', params.mail_from, '-s', subject, params.mail_recipient].execute() << """

        Pipeline execution summary
        --------------------------
        Completed at  : ${workflow.complete}
        Duration      : ${workflow.duration}
        Success       : ${workflow.success}
        workDir       : ${workflow.workDir}
        exit status   : ${workflow.exitStatus}
        Error report  : ${workflow.errorReport ?: '-'}
        """
    }
}
With
log.info 'To install mailx, run:'
log.info '  sudo apt-get install heirloom-mailx'
log.info ''
log.info 'Then create or edit the file ~/.mailrc and add
log.info '  set smtp="mail.SERVER.com:587"'
log.info '  set smtp-use-starttls'
log.info '  set smtp-auth=login'
log.info '  set smtp-auth-user="USER"'
log.info '  set smtp-auth-password="YOUR_PASSWORD"'
log.info '  set ssl-verify=ignore'
log.info ''
Phil Ewels
@ewels
Jun 14 2017 17:55
@Shellfishgene ah I see sorry, no - no transcriptome counting currently. Genome alignments for everything.
Phil Ewels
@ewels
Jun 14 2017 19:19
@fmorency - I had something very similar with mail: can you use mailx to send HTML multipart email reliably on different platforms?
Félix C. Morency
@fmorency
Jun 14 2017 19:20
I don't know about that. We use mailx because it supports TLS/SSL built-in
Afaik, mailx is a better mail
Mike Smoot
@mes5k
Jun 14 2017 19:27
Hi @pditommaso, not sure if this is by design or not, but it appears that a pipeline can have two different processes with the same name. We ran into this problem when we were merging two pipelines. Actually, it's not really a problem because both processes run as expected, it's just my expectations were challenged since I would have imagined that this would fail with some sort of duplicate name error.
Paolo Di Tommaso
@pditommaso
Jun 14 2017 19:48
Yes, it's not a problem but I understand that can be confusing. I will add a warning
Mike Smoot
@mes5k
Jun 14 2017 19:52
Warning sounds great, thanks!