These are chat archives for nextflow-io/nextflow

21st
Apr 2017
chdem
@chdem
Apr 21 2017 15:39
Hello Paolo ! I would like to have access to the mnemonic run name
Paolo Di Tommaso
@pditommaso
Apr 21 2017 15:39
what do you mean ?
chdem
@chdem
Apr 21 2017 15:40
I would like to obtain the run name like "gloomy_wilson"
or "berserk_hypatia"
from the nextflow script
I've seen in the Session.groovy some interesting variables
Paolo Di Tommaso
@pditommaso
Apr 21 2017 15:41
ahh
chdem
@chdem
Apr 21 2017 15:41
and the Mnemonic name of run instance is there
(runName)
Paolo Di Tommaso
@pditommaso
Apr 21 2017 15:41
use workflow.runName
chdem
@chdem
Apr 21 2017 15:41
great !
sorry, it is in the doc
:(
Paolo Di Tommaso
@pditommaso
Apr 21 2017 15:42
:+1:
Félix C. Morency
@fmorency
Apr 21 2017 16:12
@pditommaso can we dump de hashes (and only those) in a separate file?
Mike Smoot
@mes5k
Apr 21 2017 16:14
<cough> nextflow-io/nextflow#211 <cough>
Paolo Di Tommaso
@pditommaso
Apr 21 2017 16:15
oops
Mike Smoot
@mes5k
Apr 21 2017 16:15
:)
Paolo Di Tommaso
@pditommaso
Apr 21 2017 16:15
in the pipeline
Félix C. Morency
@fmorency
Apr 21 2017 16:37
!
Félix C. Morency
@fmorency
Apr 21 2017 17:08
@pditommaso is there an external tool (ie. md5sum) that can be used to produce the same file hash as NF?
Paolo Di Tommaso
@pditommaso
Apr 21 2017 17:08
I don't think so
NF uses murmur 128 hash implemented by google guava
Félix C. Morency
@fmorency
Apr 21 2017 17:12
Ok thanks
Félix C. Morency
@fmorency
Apr 21 2017 17:39
Will there be any advanced NF courses?
Paolo Di Tommaso
@pditommaso
Apr 21 2017 17:40
we are planning to organise an hackathon
Mike Smoot
@mes5k
Apr 21 2017 17:42
Will the hackathon be part of the September meeting?
Paolo Di Tommaso
@pditommaso
Apr 21 2017 17:42
yep
Mike Smoot
@mes5k
Apr 21 2017 17:42
Good to know!
Félix C. Morency
@fmorency
Apr 21 2017 17:43
sorry, I thought I was on the other window
Paolo Di Tommaso
@pditommaso
Apr 21 2017 17:43
:)
Mike Smoot
@mes5k
Apr 21 2017 18:00

@pditommaso I'm running into a weird problem. Here is the relevant snippet of code:

    output:
    set val("${fasta}"), file(fasta), file("${fasta}.*") into blast_db

    script:
    """
    makeblastdb -in ${fasta} -dbtype prot -input_type fasta
    """
}

blast_input = masked_fasta_files
    .combine(blast_db)
    .view{ "blast input ${it}" }

process runBlast {
    container "${blast_container}"
    tag { subseq_id }
    cache 'deep'
    cpus 8

    input:
    set val(contig_id), val(subseq_id), file(fasta),
        val(blastdb), file(blast_db_file_list) from blast_input

The problem is that in the last line the files found in file(blast_db_file_list) are not being symlinked into the work directory. I'm wondering if the fileness of the values in blast_db_file_list is getting erased somehow by combine? The output from the view operator looks like what I'd expect with a bunch of full paths listed for blast_db_file_list. Any ideas?

Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:03
the symlink aren't created or are broken ?
Mike Smoot
@mes5k
Apr 21 2017 18:03
not created
Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:03
um
I would inspect the .command.run to double check if the command to create the symlinks are correct
Mike Smoot
@mes5k
Apr 21 2017 18:05
No, I don't see them in there.
Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:06
just in case use
  input:
    set val(contig_id), val(subseq_id), file(fasta), val(blastdb), file(blast_db_file_list) from blast_input
instead of
  input:
    set val(contig_id), val(subseq_id), file(fasta),
        val(blastdb), file(blast_db_file_list) from blast_input
Mike Smoot
@mes5k
Apr 21 2017 18:07
will try
no, getting the same problem
Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:10
can you dump the content of the channel blast_input
Mike Smoot
@mes5k
Apr 21 2017 18:14
Here's what I see
blast input [we03730_nuc_v150325_PM_chr_16, we03730_nuc_v150325_PM_chr_16_blast_0, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/68/1811ce3f26d78437b08eb99e23cd87/we03730_nuc_v150325_PM_chr_16_blast_0_masked.fa, blastable.fasta, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta, [/local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.00.phr, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.00.pin, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.00.psq, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.01.phr, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.01.pin, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.01.psq, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.02.phr, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.02.pin, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.02.psq, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.03.phr, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.03.pin, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.03.psq, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.pal]]
Which looks like what I'd expect
Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:17
wait, the structure doesn't match with the input def
[
  we03730_nuc_v150325_PM_chr_16, 
  we03730_nuc_v150325_PM_chr_16_blast_0, 
  /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/68/1811ce3f26d78437b08eb99e23cd87/we03730_nuc_v150325_PM_chr_16_blast_0_masked.fa, 
  blastable.fasta, 
  /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta, 
  [/local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.00.phr, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.00.pin, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.00.psq, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.01.phr, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.01.pin, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.01.psq, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.02.phr, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.02.pin, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.02.psq, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.03.phr, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.03.pin, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.03.psq, /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/a0/5f241c72b9f5579e232a2f57ad028e/blastable.fasta.pal]
]
there are 6 elements
in your input only 5
Mike Smoot
@mes5k
Apr 21 2017 18:18
AH! Yes!
That's because my old code used to look like this:
blast_input = fasta_files
    .spread(blast_db)
    // This groups the unknown number of blastdb files into
    // a list so they can be treated as a single list of files.
    .map{ l -> tuple( l[0], l[1], l[2], l[3], l[4..l.size()-1] ) }
Félix C. Morency
@fmorency
Apr 21 2017 18:19
(can't you do just l[4..-1])?
Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:19
I think it's time to add a warning when the tuple size doesn't match the definition
Mike Smoot
@mes5k
Apr 21 2017 18:19
I changed the code becausespread started splitting the tuple in blast_db across the values in my channel instead of keeping the tuple together.
@fmorency yes, if I were smarter!
Félix C. Morency
@fmorency
Apr 21 2017 18:20
:)
Mike Smoot
@mes5k
Apr 21 2017 18:21
Thanks for your keen eye Paolo!
Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:21
:+1:
Mike Smoot
@mes5k
Apr 21 2017 18:24
Any idea why spread would have started behaving differently?
Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:26
uh? I'm not aware of that
Mike Smoot
@mes5k
Apr 21 2017 18:30
I wasn't able to reproduce what I was seeing with a test case, but I think this is the second time I've encountered spread behaving differently with 0.24.x. No worries for now - and the combine solution is much simpler - but I'll keep an eye out and try to create a test case if I encounter this behavior again.
Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:37
I don't think spread has changed, it has always flattened the result
nextflow-io/nextflow#241
Mike Smoot
@mes5k
Apr 21 2017 18:54
Right, I remember that ticket and I didn't think you'd introduce a breaking change! :) If I run anything like this again, I'll be sure to dig deeper. I've tried creating a test case to show what I was seeing, but I haven't had any luck (i.e. spread behaves just fine). In any case, thanks for the help!
Paolo Di Tommaso
@pditommaso
Apr 21 2017 18:54
ok, if you find a different behavior open an issue
Mike Smoot
@mes5k
Apr 21 2017 18:55
will do