These are chat archives for nextflow-io/nextflow

26th
Feb 2018
Paolo Di Tommaso
@pditommaso
Feb 26 2018 08:25
@danielecook weird, please create an issue attaching the .nextflow.log and nextflow.config files

wait, NF tried to execute:

qsub -N nf-kmer_countin .command.run

which is the correct PBS submit command line

but your system replied:
There was an error running the SLURM sbatch command.
The command was:
'/usr/bin/sbatch -e nf-kmer_countin.e%A -o nf-kmer_countin.o%A -J nf-kmer_countin .command.run 2>&1'
and the output was:
'sbatch: error: invalid partition specified: genomicsguest
sbatch: error: Batch job submission failed: Invalid partition name specified
I think you need to talk with your sysadmins .. :)
Luca Cozzuto
@lucacozzuto
Feb 26 2018 09:07
Hi @pditommaso are you planning a way to classify the processes to group them when specifying the configuration in nextflow config? (i.e. 5 different processes that would need the same configuration could be grouped in "bigprocesses" and other 20 ones in "smallprocesses")
Paolo Di Tommaso
@pditommaso
Feb 26 2018 09:07
this sounds an interesting idea
for resources allocation right?
Luca Cozzuto
@lucacozzuto
Feb 26 2018 09:09
yes
can I make an issue?
Paolo Di Tommaso
@pditommaso
Feb 26 2018 09:09
yep
Luca Cozzuto
@lucacozzuto
Feb 26 2018 09:09
(PS: from me only interesting ideas :) )
Paolo Di Tommaso
@pditommaso
Feb 26 2018 09:10
:)
Luca Cozzuto
@lucacozzuto
Feb 26 2018 09:16
Done #623 :)
Paolo Di Tommaso
@pditommaso
Feb 26 2018 09:16
:+1:
Luca Cozzuto
@lucacozzuto
Feb 26 2018 11:03
the same could be applied to containers... to have some processes using the container 1 and others the 2 and others the container 3 etc..
marchoeppner
@marchoeppner
Feb 26 2018 14:33
Hi, been running into some troubles with ,y Nextflow pipeline using GATK4. I am getting an error "/bin/env python - too many levels of symbolic links". When I submit the .command.sh as a standalone job my hand, it starts as expected. So I am unsure where the problem lies and whether it could be something with nextflow? Is nextflow using some kind of python virtual env or something that could explain this?
Alexander Peltzer
@apeltzer
Feb 26 2018 14:36
No it doesn't - are you using a containerized version of GATK4 ?
They might ship some python tools with the e.g. Docker containers they provide
marchoeppner
@marchoeppner
Feb 26 2018 14:37
gatk actually uses conda for individual tools, so this seems like the most obvious source of errors
but it does not happen when I run the job stand-alone, so that is certainly weird.
maybe nextflow fails to carry certain information with it that gatk conda needs? strange tho
Alexander Peltzer
@apeltzer
Feb 26 2018 14:41
Hm
I just used the official GATK4 container / docker container they provided
That worked quite well
marchoeppner
@marchoeppner
Feb 26 2018 14:48
not an option on a cluster unfortunately
(where you dont have root, that is)
ah well, back to debugging..
Alexander Peltzer
@apeltzer
Feb 26 2018 14:49
Try to get Singularity there ;-) Works fine here, on a cluster...
mgjmg
@mgjmg
Feb 26 2018 15:06
are containers for NF best (?recommended ? required) to be build including krallin/tini ??
Tintest
@Tintest
Feb 26 2018 15:49

Hello everyone,

I'm trying to do a basic QC, everything seems to work fine, except I have an input.1 file givent in argument to my fastqc process, which cause an error. Do you have any idea how to handle this temporary file created to nextflow ? How to get rid of it or just not giving it to my process ?

Thank you.

Channel
.fromFilePairs("${params.resultDir}/bcl2fastq/*_{R1,R2}*.fastq.gz",flat:true)
.filter { it[0] != "Undetermined_S0_L001" }
.filter { it[0] != "Undetermined_S0_L002" }
.set { fastq_list }


process fastqc {
    echo true
    publishDir "${params.resultDir}/fastqc", mode: 'copy'


    input:
    file fastq from fastq_list

    output:
    file '.zip' into fastq_out_zip
    file '*.html' into fastq_out_html
    stdout fastqcout

    script :

    """
    fastqc $fastq
    """
}
fastqcout.subscribe { print "$it" }
Here is the error
Failed to process input.1
uk.ac.babraham.FastQC.Sequence.SequenceFormatException: ID line didn't start with '@'
    at uk.ac.babraham.FastQC.Sequence.FastQFile.readNext(FastQFile.java:158)
    at uk.ac.babraham.FastQC.Sequence.FastQFile.<init>(FastQFile.java:89)
    at uk.ac.babraham.FastQC.Sequence.SequenceFactory.getSequenceFile(SequenceFactory.java:106)
    at uk.ac.babraham.FastQC.Sequence.SequenceFactory.getSequenceFile(SequenceFactory.java:62)
    at uk.ac.babraham.FastQC.Analysis.OfflineRunner.processFile(OfflineRunner.java:152)
    at uk.ac.babraham.FastQC.Analysis.OfflineRunner.<init>(OfflineRunner.java:121)
    at uk.ac.babraham.FastQC.FastQCApplication.main(FastQCApplication.java:316)
Paolo Di Tommaso
@pditommaso
Feb 26 2018 16:00
@marchoeppner nope, NF is virtualenv free ..
@mgjmg can use any container in which provided in include bash and gnu coretutils
@Tintest fileFromPairs produces a pair (well in your case a triple because you have specified flat:true
therefore your input declaration should be
input:
    file pair, read1, read2 from fastq_list
you should get a warning message actually
Tintest
@Tintest
Feb 26 2018 16:05
So you mean, without flat my input declaration is ok ? Or anyway I should follow yours ?
Paolo Di Tommaso
@pditommaso
Feb 26 2018 16:06
nope without flat:true it should be
input:
    file pair, reads from fastq_list
Tintest
@Tintest
Feb 26 2018 16:06
Oh ok, thanks alot ! :)
Paolo Di Tommaso
@pditommaso
Feb 26 2018 16:06
because both read files will be captured by the reads param
but in both cases the first element is the pair_id
Daniel E Cook
@danielecook
Feb 26 2018 16:19
Again... @pditommaso this does appear to by a sysadmin issue… thanks
Paolo Di Tommaso
@pditommaso
Feb 26 2018 16:19
I guess you run slurm behind the scene