These are chat archives for nextflow-io/nextflow

23rd
Oct 2018
Rad Suchecki
@rsuchecki
Oct 23 2018 03:15 UTC
Hi All. Is there a way of executing native (NF/groovy) code same as you would afterScript? I'd like to be able to read some of the files generated by the script: block and store information in a map alongside other metadata to be pass down a channel rather than throwing more tiny files into the output channel.
Paolo Di Tommaso
@pditommaso
Oct 23 2018 08:35 UTC
@rsuchecki Unfortunately no. You may want to join this thread #903. Interested to know more about your use case
@micans if you are using k8s there should not be any docker command, make sure there's docker.enabled=false in your config (I guess this is a glitch with a nf-core pipeline conf ..)
any feedback to improve it is welcome
Maxime Garcia
@MaxUlysse
Oct 23 2018 09:01 UTC
\o/
It looks amazing
I need to try that now
Alexander Peltzer
@apeltzer
Oct 23 2018 09:06 UTC
I also like it quite a lot
Playing around with it a little atm too
Paolo Di Tommaso
@pditommaso
Oct 23 2018 09:07 UTC
I think, it can be improved but at the same time don't want to add too many info
Luca Cozzuto
@lucacozzuto
Oct 23 2018 09:08 UTC
wow
Paolo Di Tommaso
@pditommaso
Oct 23 2018 09:11 UTC
:joy:
Stijn van Dongen
@micans
Oct 23 2018 09:54 UTC
:+1: Thanks @pditommaso for docker.enabled=false. That may well be it (now testing). It was set to true in our pipeline (globally), but only made a difference for multiqc somehow.
Mmmm. I now get multiqc not found. This must be simple ... more coffee needed.
Paolo Di Tommaso
@pditommaso
Oct 23 2018 10:14 UTC
that was a problem that in some nfcore pipeline the container was not specified for the multiqc step
Stijn van Dongen
@micans
Oct 23 2018 11:01 UTC
I haven't had coffee yet (back from meeting). But we run both multiqc and fastqc on k8s using docker images in our rnaseq pipeline. For testing I've made pods with both multiqc and fastqc, I can log into them, and both softwares exist in /usr/local/bin/ and work. Fastqc works always (regardless of docker.enabled setting), multiqc fails with 'docker not found' if docker.enabled = true and it fails with 'multiqc not found' if docker.enabled = false
Tagging @ewels as well.
Alexander Peltzer
@apeltzer
Oct 23 2018 11:29 UTC
Hes on vacation ;-)
Krittin Phornsiricharoenphant
@sinonkt
Oct 23 2018 11:49 UTC
Hi, guys, when submit job on real HPC cluster, What i need to do is login to frontend node then run nextflow run my_flow.nf as background process, right? if i need multiple of them i just forked nf-process and let them run as background on this frontend til they exit?
is these practical if the each nf-background-process takes days to run and stay on that frontend node?
Paolo Di Tommaso
@pditommaso
Oct 23 2018 11:58 UTC
Yes, the driver app is not compute intensive, you can have it to run in background in the login node
Alternatively you can launch it as a cluster job as we
*well
Krittin Phornsiricharoenphant
@sinonkt
Oct 23 2018 11:59 UTC
like sbatch nextflow run something like that?
Paolo Di Tommaso
@pditommaso
Oct 23 2018 12:00 UTC
Ask your sys admin what's the best for your cluster
Krittin Phornsiricharoenphant
@sinonkt
Oct 23 2018 12:00 UTC
@pditommaso Thank you so much :)
Paolo Di Tommaso
@pditommaso
Oct 23 2018 12:01 UTC
:+1:
Stijn van Dongen
@micans
Oct 23 2018 12:39 UTC
Thanks @apeltzer
Stijn van Dongen
@micans
Oct 23 2018 13:11 UTC

Allright, I found it ... it lived in our conf/base.config that came from nf-core/rnaseq. It has this:

  withName: multiqc {
    executor = 'local'
  }

After reading this old issue nextflow-io/nextflow#693 it prompted me to search for local executors ... there it was in the base config.

Paolo Di Tommaso
@pditommaso
Oct 23 2018 13:43 UTC
exactly that's the problem
Crabime
@Crabime
Oct 23 2018 14:43 UTC
Hi Paolo, whether can i move my work directory to another host and resume?
Paolo Di Tommaso
@pditommaso
Oct 23 2018 14:44 UTC
no
Crabime
@Crabime
Oct 23 2018 14:44 UTC
I think if i copy .nextflow files the same parent folder with work directory and my nextflow executable file and $HOME/.nextflow folder to such host
then it should be ok to resume previous job
Paolo Di Tommaso
@pditommaso
Oct 23 2018 14:47 UTC
as you wish ..
Crabime
@Crabime
Oct 23 2018 14:50 UTC
i already tried but failed, i found it return different HashCode for the same process, and it wouldn't if resume in the same host
Stijn van Dongen
@micans
Oct 23 2018 15:31 UTC
@Crabime I'm interpreting Paolo's answer as that it is not possible. Probably best to do the same. I assume hash code may use the full path, not a relative path. Doesn't matter though if the answer is a clear no!
Tobias "Tobi" Schraink
@tobsecret
Oct 23 2018 15:39 UTC
Hi folks, I have some trouble with a download script. It downloads all fastq.gz files for a given ENA accession. Unfortunately it seems sometimes it does not throw an error if terminated prematurely. For example it will only download 3 fastq.gz files for an accession with two paired end samples, for which there are 4 fastq.gz files. How can I force it to rerun if there is an uneven number of output files?
process download_reads {
    /*
    This process downloads fastq.gz files from ENA accessions. I run enaDataGet for accessions which point to
    only one sequencing lane, i.e. one pair of fastq files. It fails for accessions that point to multiple
    lanes, so in that case I use enaGroupGet instead. Both of these utilities are supplied by ENA:
    https://github.com/enasequence/enaBrowserTools
    */
    tag "$accession"
    publishDir 'reads', mode: 'symlink'
    cpus 1
    time '1h'
    errorStrategy 'retry'
    maxRetries 3
    memory {8.GB * task.attempt}
    time {1.hour * task.attempt}


    input:
    val accession from accessions

    output:
    set accession, file('**.fastq.gz') into downloaded_reads

    script:
    """
    enaDataGet -f fastq $accession || enaGroupGet -f fastq $accession
    """

}
Paolo Di Tommaso
@pditommaso
Oct 23 2018 15:41 UTC
some ls | wc -c etc one-liner that returns a non-zero if one or more files are missing ?
Tobias "Tobi" Schraink
@tobsecret
Oct 23 2018 17:03 UTC
oh right, find . -name '*.fastq.gz' | wc -l should do the trick
Vladimir Kiselev
@wikiselev
Oct 23 2018 22:42 UTC