These are chat archives for nextflow-io/nextflow

10th
May 2017
Anthony Underwood
@aunderwo
May 10 2017 08:20
@pditommaso just seen the blog post for workshop in September. Got straight on and applied!
Shellfishgene
@Shellfishgene
May 10 2017 08:26
Which nextflow video?
Paolo Di Tommaso
@pditommaso
May 10 2017 08:51
@aunderwo well done
Anthony Underwood
@aunderwo
May 10 2017 08:52
@pditommaso Just hope I make the grade with the selection :)
Paolo Di Tommaso
@pditommaso
May 10 2017 08:52
@Shellfishgene I don't know exactly which video he was referring, however there are some videos here
@aunderwo I guess you will ;)
mitul-patel
@mitul-patel
May 10 2017 11:15
Hello, is it possible to use bash variable outside process ?
Paolo Di Tommaso
@pditommaso
May 10 2017 12:07
Do you mean this nextflow-io/nextflow#69 ?
mitul-patel
@mitul-patel
May 10 2017 12:55
here is the code:
        process index { 

        publishDir "$outDir", mode:'copy', overwrite: true

        input:
       file fasta from genome

        script:
        """
        bowtie2-build --threads 8 ${fasta} genome.index
        mode="index"
        """ 
     }

      if ('index' in "\${mode}") {
                     ...............
     }
Paolo Di Tommaso
@pditommaso
May 10 2017 12:56
so the question is ?
mitul-patel
@mitul-patel
May 10 2017 13:08
is it possible to use mode outside the process index...mode is defined inside script block of process index.
Paolo Di Tommaso
@pditommaso
May 10 2017 13:11
no, it's not possible
said that you can declare it outside the bash script, use in the bash script and then output it
eg
output: 
val mode into something_ch 
 script:
        def  mode="index"
        """
        bowtie2-build --threads 8 ${fasta} genome.index
        echo $mode
        """
mitul-patel
@mitul-patel
May 10 2017 13:16
Thanks @pditommaso. It will solve my problem...
Paolo Di Tommaso
@pditommaso
May 10 2017 13:16
good
Maxime Garcia
@MaxUlysse
May 10 2017 13:38
Hello @pditommaso do you have a vector version of the nextflow logo by any chance ?
Paolo Di Tommaso
@pditommaso
May 10 2017 13:38
not a vector version but an hi-res one for sure
just a sec
Maxime Garcia
@MaxUlysse
May 10 2017 13:39
Thanks a lot
Paolo Di Tommaso
@pditommaso
May 10 2017 13:49
I've uploaded it in this repo
Maxime Garcia
@MaxUlysse
May 10 2017 13:53
Perfect, I indeed had a low-res version
Robert Syme
@robsyme
May 10 2017 13:58
I don't know when the combine operator was introduced, but it is fantastic. Thanks all!
Paolo Di Tommaso
@pditommaso
May 10 2017 13:59
Hi Rob, how long!
Yes, combine has been introduced a couple of months ago .. under the suggestion of @skptic
Alessia
@alesssia
May 10 2017 14:11

Hello, I have a small problem that I already know will turn out to be very stupid, but I can figure it out :sob: I am submitting a process with this configuration:

executor = 'pbs'
queue = 'pqeph'

$qualityAssessmentRaw 
{
    clusterOptions = "-l select=1:ncpus=4:mem=8G"            
    time =  '15m'
    jobName = "qualityAssessmentRaw"      
}

but the .command.run file has the following header

#!/bin/bash
#PBS -N nf-qualityAsses
#PBS -o ~/9d/19094b82ae46a2723a3a442cc7bf8d/.command.log
#PBS -j oe
#PBS -V
#PBS -q pqeph
#PBS -l nodes=1:ppn=4
#PBS -l walltime=00:15:00
#PBS -l mem=8gb
#PBS -l select=1:ncpus=4:mem=8G

who does specify #PBS -l nodes=1:ppn=4 and #PBS -l mem=8gb? These directives are forbidden by the system...

Paolo Di Tommaso
@pditommaso
May 10 2017 14:13
remove cpus and memory in your config
you will need to handle manually
Alessia
@alesssia
May 10 2017 14:13
they are not there
Paolo Di Tommaso
@pditommaso
May 10 2017 14:14
what is the output of nextflow config command ?
Alessia
@alesssia
May 10 2017 14:16
the same file I use as nextflow.config file, with cpu/memory are only mentioned as clusterOptions='-l select=1:ncpus=4:mem=8G' do you want to see it all?
Paolo Di Tommaso
@pditommaso
May 10 2017 14:16
yes
Paolo Di Tommaso
@pditommaso
May 10 2017 14:21
are you sure you haven't put cpus or memory in the main pipeline script ?
Alessia
@alesssia
May 10 2017 14:22
I am (also grep could not find them)
Paolo Di Tommaso
@pditommaso
May 10 2017 14:23
if so, NF won't produce the #PBS -l nodes=1:ppn=4
Alessia
@alesssia
May 10 2017 14:23
but it does :)
Paolo Di Tommaso
@pditommaso
May 10 2017 14:23
it's not possible :)
Alessia
@alesssia
May 10 2017 14:23
I swear!
Paolo Di Tommaso
@pditommaso
May 10 2017 14:23
um, in your snippet the process name is #PBS -N nf-qualityAsses
while in the config is qualityAssessmentRaw
Alessia
@alesssia
May 10 2017 14:24
it cuts the name that should be qualityAssessmentRaw
Paolo Di Tommaso
@pditommaso
May 10 2017 14:24
something is not matching
Alessia
@alesssia
May 10 2017 14:24
all the names are cut
but when running
[37/3b0bef] Submitted process > qualityAssessmentRaw (2)
[df/b1f1ee] Submitted process > qualityAssessmentRaw (1)
Shellfishgene
@Shellfishgene
May 10 2017 14:25
The pbs executor cuts the names to 15 characters
Paolo Di Tommaso
@pditommaso
May 10 2017 14:25
ah, good point
Alessia
@alesssia
May 10 2017 14:25
Yep, this is what I mean with all are cut!
any other place where memory/cpu could be specified?
Paolo Di Tommaso
@pditommaso
May 10 2017 14:27
config file(s) and pipeline script
Alessia
@alesssia
May 10 2017 14:27
:sob:
Paolo Di Tommaso
@pditommaso
May 10 2017 14:28
don't cry, there should be a reason :)
can you upload the .nextflow.log file ?
at the top there's the list of parsed config files
Alessia
@alesssia
May 10 2017 14:28
I've been looking into this for 1h now! It's empty it is rejected by the queue
Paolo Di Tommaso
@pditommaso
May 10 2017 14:31
what about the log file ?
Alessia
@alesssia
May 10 2017 14:31
what other log?
Paolo Di Tommaso
@pditommaso
May 10 2017 14:32
.nextflow.log file
in the launching dir
Alessia
@alesssia
May 10 2017 14:32
right
nothing useful
same things it says already on screen

May-10 15:32:22.895 [Actor Thread 4] ERROR nextflow.processor.TaskProcessor - Error executing process > 'qualityAssessmentRaw (1)'

Caused by:
Failed to submit job to grid scheduler for execution

Command executed:

qsub -N nf-qualityAsses .command.run

Command exit status:
6

Command output:
qsub: Cannot be used with select or place: nodes

May-10 15:32:22.900 [Actor Thread 4] DEBUG nextflow.Session - Session aborted -- Cause: Error submitting process 'qualityAssessmentRaw (1)' for execution
Paolo Di Tommaso
@pditommaso
May 10 2017 14:35
at the beginning there's the list of config files parsed
how many they are ?
Alessia
@alesssia
May 10 2017 14:36
one
the one I am expecting it will use
Paolo Di Tommaso
@pditommaso
May 10 2017 14:36
I see.
but already run a NF script in your cluster in the past, right ?
Alessia
@alesssia
May 10 2017 14:37
ok, nevermind. It must be something absolutely obvious thus impossible to spot
Paolo Di Tommaso
@pditommaso
May 10 2017 14:38
at the beginning of your config I see
env {
    threads=4
    maxmem='32G'
}
Alessia
@alesssia
May 10 2017 14:38
yeah, I did, and was fine until they changed the scheduler directives abolishing those. At the start we got warnings, so fixed it and worked fine. On Monday they have been enforced and now is broken
yep, they are used for BBmap
within one of the process
bbduk.sh -Xmx$maxmem
Paolo Di Tommaso
@pditommaso
May 10 2017 14:39
I don't know. I'm unable to replicate your issue.
Alessia
@alesssia
May 10 2017 14:39
ok, nevermind. I will be back on this with fresh mind
Paolo Di Tommaso
@pditommaso
May 10 2017 14:39
try in your local machine
Alessia
@alesssia
May 10 2017 14:40
it works
thanks anyway! :)
Paolo Di Tommaso
@pditommaso
May 10 2017 14:40
welcome
Alessia
@alesssia
May 10 2017 14:40
(I mean, it works on my machine)
Paolo Di Tommaso
@pditommaso
May 10 2017 14:41
yes, but if you specify the pbs executor the wrong directive is produced ?
Alessia
@alesssia
May 10 2017 14:42
I think it is? I don't remember I've done some many tests. Will look, good point
Paolo Di Tommaso
@pditommaso
May 10 2017 14:42
ok
Alessia
@alesssia
May 10 2017 14:45
thanks :)