These are chat archives for nextflow-io/nextflow

16th
Oct 2017
Simone Baffelli
@baffelli
Oct 16 2017 09:00 UTC
Good morning.
How is it possible that these (commented!!) lines invalidate a process cache?
// //Convert phase to meters
// phaseToMeters = params.radar_frequency / (2 * Math.PI) 

// //Convert seconds to day
// def secondsToDay(Float bl)
// {
//     return bl / (60 * 60 * 24)
// }
They are defined on top of my pipeline
Paolo Di Tommaso
@pditommaso
Oct 16 2017 09:01 UTC
script?
Simone Baffelli
@baffelli
Oct 16 2017 09:01 UTC
what do you mean?
Paolo Di Tommaso
@pditommaso
Oct 16 2017 09:02 UTC
where's that code ?
Simone Baffelli
@baffelli
Oct 16 2017 09:14 UTC
the one I posted above
//Convert phase to meters
phaseToMeters = params.radar_frequency / (2 * Math.PI) 

//Convert seconds to day
 def secondsToDay(Float bl)
 {
     return bl / (60 * 60 * 24)
 }
Paolo Di Tommaso
@pditommaso
Oct 16 2017 09:15 UTC
I mean, where is placed that code ?
Simone Baffelli
@baffelli
Oct 16 2017 09:15 UTC
on top of the pipeline
before channels and processes are defined
Paolo Di Tommaso
@pditommaso
Oct 16 2017 09:15 UTC
not in a process ?
Simone Baffelli
@baffelli
Oct 16 2017 09:16 UTC
no
Paolo Di Tommaso
@pditommaso
Oct 16 2017 09:16 UTC
if so it should not alter the caching
Simone Baffelli
@baffelli
Oct 16 2017 09:16 UTC
i agree
but it does!
Even when commented
Paolo Di Tommaso
@pditommaso
Oct 16 2017 09:17 UTC
create a replicable test case and open an issue :)
Simone Baffelli
@baffelli
Oct 16 2017 09:18 UTC
thats the point :)
how to make it replicable
Paolo Di Tommaso
@pditommaso
Oct 16 2017 09:19 UTC
welcome in our fantastic world ..
Simone Baffelli
@baffelli
Oct 16 2017 09:19 UTC
ahaha
:snail:
Paolo Di Tommaso
@pditommaso
Oct 16 2017 09:19 UTC
I'm going crazy with a bug happing ONLY when running in travis ci
Simone Baffelli
@baffelli
Oct 16 2017 09:20 UTC
does it use different versions of something?
like different versions of the same class?
Paolo Di Tommaso
@pditommaso
Oct 16 2017 09:20 UTC
nope
Simone Baffelli
@baffelli
Oct 16 2017 09:20 UTC
weird
Simone Baffelli
@baffelli
Oct 16 2017 09:49 UTC

Appartently this piece of code is very malicious:

// //Convert seconds to day
// def secondsToDay(Float bl)
// {
//     return bl / (60 * 60 * 24)
// }

But I don't understand why

Simone Baffelli
@baffelli
Oct 16 2017 09:56 UTC
It is especially weird because the code is commented
Paolo Di Tommaso
@pditommaso
Oct 16 2017 11:55 UTC
Just roll out a new beta 0.26.0-beta5, it fixes some bugs and introduces the support for fastq compression and loong await paired-end splitting
for example
Luca Cozzuto
@lucacozzuto
Oct 16 2017 11:56 UTC
what is fastq compression?
Paolo Di Tommaso
@pditommaso
Oct 16 2017 11:56 UTC
Channel
       .fromFilePairs('/some/path/*_{1,2}.fq.gz')
       .splitFastq(compress:true, file:true, pe: true)
Luca Cozzuto
@lucacozzuto
Oct 16 2017 11:56 UTC
wonderful
Paolo Di Tommaso
@pditommaso
Oct 16 2017 11:57 UTC
gzip compression for fastq chunks
Simone Baffelli
@baffelli
Oct 16 2017 11:57 UTC
:confused: I'm going crazy
Adding or removing some code at the beginning of the pipeline messes the caching
and I cannot reproduce it
Paolo Di Tommaso
@pditommaso
Oct 16 2017 11:58 UTC
if you cannot reproduce it *may* not be the problem
Simone Baffelli
@baffelli
Oct 16 2017 11:59 UTC
but it appears to be very consistent
Paolo Di Tommaso
@pditommaso
Oct 16 2017 12:00 UTC
enables hashes dump and verify why they change
Simone Baffelli
@baffelli
Oct 16 2017 12:00 UTC
I'm trying to
but it is very diffcult to find them
will try with grep
Ali Al-Hilli
@Ali_Hilli_twitter
Oct 16 2017 14:25 UTC

I have trouble with submitting nextflow to some of the nodes in our SGE cluster
My nextflow.config looks like this:

profiles {

    standard {
        process.executor = 'local'
    }

    cluster {
        process {
            executor = 'sge'
            queue = 'researchsmall.q'
        }
    }
}

Queue researchsmall.q submits the jobs to a few nodes. We find that nextflow jobs work in some nodes and error out when submitted to other nodes. Tests done show that there is nothing wrong with the script but with the submission to the node. Other qsub scripts work in the nodes that error out with nextflow jobs.

The error from nextflow says:

/etc/profile.d/modules.sh: line 12: MANPATH: unbound variable

In my nextflow script before I load the modules I source /etc/profile.d/modules.sh to add MANPATH=/usr/man:/usr/share/man:/usr/X11R6/man as env variable needed for module load. This seems to fail in some nodes and succeed in other nodes.
In the nodes that fail, once I log directly into the node and source /etc/profile.d/modules.sh it works fine, its just when submitted in a nextflow script using -profile cluster then it fails and gives the above error message.
Any ideas why the nextflow script is failing in some nodes and succeed in others? This is a specific nextflow problem as other qsub scripts work fine in all our nodes.
This is my nextflow process that is failing:

process qa_and_trim {

    publishDir "${output_dir}/${pair_id}", mode: 'copy'

    // module 'phe/qa_and_trim'

    input:
    set pair_id, file(file_pair) from read_pairs
  val(workflow) from workflow

  output:
  // set val(output_dir) from params.output_dir
  file('qa_and_trim')
  set pair_id, file('*.processed.R*.fastq.gz') into processed_fastqs_channel

  //  make dir only once for the ${output_dir}/${pair_id}

  // yourString.replace(".*", "")

  script:

    """
  . /etc/profile.d/modules.sh
    module purge
     module use /phengs/hpc_software/Modules/production
     module load phe/qa_and_trim
    mkdir -p ${output_dir}/${pair_id}
    qa_and_trim.py workflow $workflow \$PWD

    """
}
Paolo Di Tommaso
@pditommaso
Oct 16 2017 14:28 UTC
this happens because NF execute your script adding a set -u directive ie. does not allow undefined environment variables
eventually this is a problem with your cluster configuration because it appears in some nodes that var is nor defined
you can try to bypass this check by adding the following setting in the nextflow config file
process.shell = ['/bin/bash','-e']
Evan Floden
@evanfloden
Oct 16 2017 14:32 UTC

Maybe more singularity/docker relatated, but when I specify in nextflow.config for singularity to do the conversion to an img:

container = 'docker://mydocker/image'

I notice that the following occurs in the background:

Cache folder set to /nfs/users2/cn/efloden/.singularity/docker
[9/9] |===================================| 100.0% 
WARNING No environment, labels files, or runscript will be included.
Creating container runtime...

So I don't have any ENV commands from docker carried over etc.

Paolo Di Tommaso
@pditommaso
Oct 16 2017 14:34 UTC
we cannot do anything here, that is how the docker image conversion is implemented by the singularity folks
Ali Al-Hilli
@Ali_Hilli_twitter
Oct 16 2017 14:35 UTC
@pditommaso ``` process.shell = ['/bin/bash','-e']
doesnt work
Paolo Di Tommaso
@pditommaso
Oct 16 2017 14:36 UTC
same problem ?
Evan Floden
@evanfloden
Oct 16 2017 14:37 UTC
@pditommaso thanks
Ali Al-Hilli
@Ali_Hilli_twitter
Oct 16 2017 14:38 UTC
@pditommaso well this time it just bombs out with the following log:
N E X T F L O W  ~  version 0.25.1
Launching `test.nf` [happy_hodgkin] - revision: 661ba5bdff
[warm up] executor > sge
[0d/90058e] Submitted process > qa_and_trim (1)
ERROR ~ Error executing process > 'qa_and_trim (1)'

Caused by:
  Process `qa_and_trim (1)` terminated with an error exit status (138)

Command executed:

  source /etc/profile.d/modules.sh
  module purge
      module use /phengs/hpc_software/Modules/production
      module load phe/qa_and_trim
  mkdir -p /phengs/hpc_storage/run_data/hiseq/D00151/170405_D00151_0346_BHHKK5BCXY/nextflow_output/358377_H17106021811-2700-2.staphylococcus-aureus-typing.ngsservice
  qa_and_trim.py workflow staphylococcus-aureus-typing.ngsservice $PWD

Command exit status:
  138

Command output:
  (empty)

Work dir:
  /phengs/hpc_storage/home/ngsservice/scripts/nextflow/work/0d/90058ec2317e40e45368c03ee3e855

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
no new info
Paolo Di Tommaso
@pditommaso
Oct 16 2017 14:39 UTC
umm .. error 138 looks like an error reported by SGE
change into /phengs/hpc_storage/home/ngsservice/scripts/nextflow/work/0d/90058ec2317e40e45368c03ee3e855
and execute qsub .command.run and verify why is not executed
eventually using qstat -j <jobid> to get more info
Ali Al-Hilli
@Ali_Hilli_twitter
Oct 16 2017 15:08 UTC
It looked like nextflow was unsetting the env variables so we changed the modules.sh to check if MANPATH is unset and if unset then set it. This works now, thanks
Paolo Di Tommaso
@pditommaso
Oct 16 2017 15:08 UTC
:+1:
also note you don't need to create mkdir -p ${output_dir}/${pair_id}
Ali Al-Hilli
@Ali_Hilli_twitter
Oct 16 2017 15:31 UTC
@pditommaso I do because I will be running multiple samples and I need it to create a dir for every sample
Paolo Di Tommaso
@pditommaso
Oct 16 2017 15:33 UTC
I saw that you are using the same as publishDir "${output_dir}/${pair_id}", mode: 'copy'
hence NF will created that folder automatically
Félix C. Morency
@fmorency
Oct 16 2017 15:55 UTC
Is it possible to access a groovy map from a bash script:?
Anthony Underwood
@aunderwo
Oct 16 2017 15:59 UTC
@pditommaso So are you saying that publishDir "${output_dir}/${pair_id}"should in effect do a mkdir -p ${output_dir}/${pair_id}??
Paolo Di Tommaso
@pditommaso
Oct 16 2017 16:03 UTC
yes
basically you don't need to take create of the creation of final outputs
@fmorency in the script: you can use any groovy statement/code, the important thing is the the last statement evaluates to the command string
Félix C. Morency
@fmorency
Oct 16 2017 16:08 UTC
Ok. I guess I'll have to bridge groovy maps to bash maps
Félix C. Morency
@fmorency
Oct 16 2017 18:25 UTC
Works, thanks.
amacbride
@amacbride
Oct 16 2017 23:36 UTC

Wow, I was just going to ask a question, and it looks like you've all just been talking about something similar. Here goes:

At the moment, I have a need to save intermediate files (generated BAM and VCF files, etc.) -- I'm assuming I want to use publishDir with mode 'copy'. The pipeline itself properly using input and output declarations for all inputs and outputs, and doesn't depend on files being in the publishDir.

I'm assuming that all the intermediate staging into the scratch directory is done by symlinking files, and is essentially separate? (As in, the fact that the output of Process A is being copied to the publishDir asynchronously won't affect the fact that Processes B, C, and D that depend on the output of A might be running concurrently?) @pditommaso