These are chat archives for nextflow-io/nextflow

5th
Jun 2018
Maxime HEBRARD
@mhebrard
Jun 05 2018 04:48
hello .. someone around here ?
I don't get it ... I try to run my workflow within singularity image ...
when I do nextflow run user/repo it pull me an image.img.Sf3lqh as the dl didnt finished !
if I run manually the command singularity pull --name the dl works well :(
Maxime HEBRARD
@mhebrard
Jun 05 2018 04:55
I was with singularity 2.4.0 and I had the bug of .img / .simg ... I upgrade with 2.5.1 and then that :(
Paolo Di Tommaso
@pditommaso
Jun 05 2018 06:47
are you referring this nextflow-io/nextflow#724 ?
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:13
sorry ... yes so .. I get the issue #724 .... then I upgrade singularity version
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:14
and ?
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:14
trying with 2.5.1 and 2.4.2 ... the problem of .simg is solved
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:15
:ok_hand:
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:15
but nextflow still failed to execute the workflow inside the image
:(
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:15
well, this is another problem
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:16
yes that is why I close the issue #724
I am trying to understand the new issue ... but I don't get consistent behavior ... it failed everytime but donno why lol
basically I always get the error from nextflow "Failed to pull singularity image"
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:18
are you trying to pull a singularity image from within a singularity container ?
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:19
hmm I pull from singularity hub
shub://mhebrard/NextFlow
on singularity-hub the build is "complete" ... if I run singularity pull ... I get the good image
inside nextflow sometimes I get a file with wrong name (.img.arg7gt like temporary file) and on the last run I get the good file but still get the error :-/
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:25
If I manually singularity pull the image, then nextflow run -with-singularity myImage.img that works .... I don't get wht the auto-pull failed :-/
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:29
how the container is specified in the nexflow.config file ?
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:29
hmm one strange thing I notice is the variable workflow.revision looks different in nextflow.config and main.nf ... I wonder if there is a conflics with the revision id between the pull command and the run command ?? (just random guess)
process.container = 'shub://mhebrard/TrimFlow:latest' singularity.enabled = true
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:32
what is the error message ?
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:32
Failed to pull singularity image command: singularity pull --name mhebrard-TrimFlow.img shub://mhebrard/TrimFlow > /dev/null status : 143
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:33
triple `
new-line
code
new-line
triple `
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:34
 Failed to pull singularity image
  command: singularity pull --name mhebrard-TrimFlow.img shub://mhebrard/TrimFlow:latest > /dev/null
  status : 143
(oh I see)
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:34
:+1:
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:35
(sorry quite new here ^^")
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:35
no pb
inside nextflow sometimes I get a file with wrong name (.img.arg7gt like temporary file)
what does it mean inside nextflow ?
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:36
let me find the example

when I run:

nextflow run mhebrard/TrimFlow --reads 'data/*_R{1,2}*'

I get the error above and if I look the pulled image

ls work/singularity
>mhebrard-TrimFlow-latest.img.FhaoBm
(for example)
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:43
is the error immediate ?
also there should be a message field after status in the error reported.
what is saying ?
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:46

hmm not immediate... I have the trace

Pulling Singularity image shub://mhebrard/TrimFlow [cache /home/vagrant/TrimFlow/work/singularity/mhebrard-TrimFlow.img]

then it wait a while (for the download to proceed I guess), then

ERROR ~ Error executing process > 'trim_galore (AMH004-TCCTGAGC-AGAGGATA_S7_L001                                   )'

Caused by:
  Failed to pull singularity image
  command: singularity pull --name mhebrard-TrimFlow.img shub://mhebrard/TrimFlo                                   w > /dev/null
  status : 143
  message:
the message field is empty :(
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:47
if takes more than 10 minutes, it's killed by NF
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:47
so 'trim_galore...' is my first process (just after the channel def)
oh so you think the problem is bcz the image take too long to download ? then the process is automatically killed ?
(I admit I do not have good connection :'( )
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:49
I don't know, BUT if it takes > 10minutes YES, else NO
:)
in the first case you can increase the timeout adding
singularity.pullTimeout = '20 mins'
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:50
well that would explain why sometime I get those weird extentions ...
nice
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:50
but it looks strange, I've downloaded it in a few seconds
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:50
does it works for you ?
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:51
only tried singulairty pull
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:51
ok ... the pullTimeout goes in the nextflow.config ?
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:52
yep
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:52
I will try then
that would be a great news ;p
Not a valid duration value: 20 mins
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:56
sorry 20 min without s
Maxime HEBRARD
@mhebrard
Jun 05 2018 07:58
sorry to ask , didnt find in the doc ^^"
Paolo Di Tommaso
@pditommaso
Jun 05 2018 07:58
no pb
Maxime HEBRARD
@mhebrard
Jun 05 2018 08:09
@pditommaso well... 11min later... the pipeline run without problem >.<
Paolo Di Tommaso
@pditommaso
Jun 05 2018 08:09
:+1:
Maxime HEBRARD
@mhebrard
Jun 05 2018 08:09
I am not sure if I should laugh or cry ...
anyway sorry to waste your afternoon ... thanks for the help :)
nextflow is awesome ;D
Paolo Di Tommaso
@pditommaso
Jun 05 2018 08:10
cool
no problem, anyhow it's 10 am in spain ;)
Maxime HEBRARD
@mhebrard
Jun 05 2018 08:12
oh ! then sorry to have wasted your morning :p
Paolo Di Tommaso
@pditommaso
Jun 05 2018 08:12
:joy:
are you in Andreas team ?
Maxime HEBRARD
@mhebrard
Jun 05 2018 08:13
hmm no, who is Andreas ?
Paolo Di Tommaso
@pditommaso
Jun 05 2018 08:13
sorry, I was thinking you were from Singapore Genome Institute
Maxime HEBRARD
@mhebrard
Jun 05 2018 08:14
oh ! GIS .. well we are neighbor ... I donno much people here yet
Paolo Di Tommaso
@pditommaso
Jun 05 2018 08:15
nice that NF community is expanding in Singapore
Maxime HEBRARD
@mhebrard
Jun 05 2018 08:15
:+1:
well I guess the person that motivate me to work with Nextflow is working with him haha
Paolo Di Tommaso
@pditommaso
Jun 05 2018 08:20
:+1:
Luca Cozzuto
@lucacozzuto
Jun 05 2018 09:24
Dear all I have a question on how to use watchPath...
is someone using it?
Is there a way to "convert" a .fromFilePairs( params.mzlfiles, size: 1) to watchPath?
Luca Cozzuto
@lucacozzuto
Jun 05 2018 09:45
@pditommaso?
Paolo Di Tommaso
@pditommaso
Jun 05 2018 09:56
umm, yes, applying a groupTuple and maybe something else
Luca Cozzuto
@lucacozzuto
Jun 05 2018 09:58
I was thinking something like map?
Paolo Di Tommaso
@pditommaso
Jun 05 2018 09:59
yes example, fromFilePairs is essentially a fromPath.map { }.groupTuple ()
Luca Cozzuto
@lucacozzuto
Jun 05 2018 10:01
can you write it better?
:)
Paolo Di Tommaso
@pditommaso
Jun 05 2018 10:02
try it, you will improve you skill ;)
Luca Cozzuto
@lucacozzuto
Jun 05 2018 10:02
-2 coffees
Paolo Di Tommaso
@pditommaso
Jun 05 2018 10:03
:joy:
Luca Cozzuto
@lucacozzuto
Jun 05 2018 10:38
Channel
       .fromPath( params.mzlfiles )             
       .ifEmpty { error "Cannot find any file matching: ${params.mzlfiles}" }
       .map { 
        file = it
        id = it.getName()
        ext = params.mzlfiles.tokenize( '/' )
        len = ext[-1].length()
        [id[0..-len], file ]
    }
    .set { mychannel}    

mychannel.println()
it is really ugly...
Alexander Peltzer
@apeltzer
Jun 05 2018 12:12
Congrats @pditommaso on 0.30.0!
Paolo Di Tommaso
@pditommaso
Jun 05 2018 12:13
I didn't announce it yet! :joy:
Alexander Peltzer
@apeltzer
Jun 05 2018 12:15
Saw the release and made one for archlinux already :-P https://aur.archlinux.org/packages/nextflow/
(call me addicted, maybe ;-))
Paolo Di Tommaso
@pditommaso
Jun 05 2018 12:16
wow! definitely ;)
Sven F.
@sven1103
Jun 05 2018 12:18
:clap:
Shellfishgene
@Shellfishgene
Jun 05 2018 12:30
Hi! I installed FastQC via bioconda, and it runs fine from the console. However with nextflow I always get "fonts not found", which apparenlty happens when a wrong java version is used. I don't get why this is only a problem when nextflow runs it. If I run a command from the .command.sh manually it works as expected...
Paolo Di Tommaso
@pditommaso
Jun 05 2018 12:31
not a NF issue, that 's a mess in the bioconda recipe for fastqc
Shellfishgene
@Shellfishgene
Jun 05 2018 12:32
But it works find outside of nf, and that's the same fastqc
Will nf use the default java that's in the path, the conda one in my case?
Paolo Di Tommaso
@pditommaso
Jun 05 2018 12:33
I had the same problem and I've fixed with this
Shellfishgene
@Shellfishgene
Jun 05 2018 12:35
Hmm, I'll try the first parts but don't have sudo to install fonts
Paolo Di Tommaso
@pditommaso
Jun 05 2018 12:35
I'm referring only to the last line
Shellfishgene
@Shellfishgene
Jun 05 2018 12:36
But why would FastQC work fine on the command line if the fonts are missing?
now we can celebrate :beers: :beers: :beers:
micans
@micans
Jun 05 2018 12:55
:fireworks: :fireworks: :fireworks: :beers:
Sven F.
@sven1103
Jun 05 2018 12:59
:tada:
congratz @pditommaso and all contributors
Paolo Di Tommaso
@pditommaso
Jun 05 2018 13:00
you are the next one :smile:
Sven F.
@sven1103
Jun 05 2018 13:00
yaaaah, I hope so :D
Yasset Perez-Riverol
@ypriverol
Jun 05 2018 13:00
are biocontainers also supported @pditommaso
?
Paolo Di Tommaso
@pditommaso
Jun 05 2018 13:00
they have always been supported
Yasset Perez-Riverol
@ypriverol
Jun 05 2018 13:01
ok cool..
Paolo Di Tommaso
@pditommaso
Jun 05 2018 13:01
there was only a problem when NF was trying to fetch job stats, because it was using ps not compatible with the one in the biocontainers base image
Yasset Perez-Riverol
@ypriverol
Jun 05 2018 13:02
ok
Félix C. Morency
@fmorency
Jun 05 2018 13:53
Congratz all for the new release!
Paolo Di Tommaso
@pditommaso
Jun 05 2018 13:54
:tada:
is conda used in neuro-imaging ?
Félix C. Morency
@fmorency
Jun 05 2018 13:55
not that I know of
Paolo Di Tommaso
@pditommaso
Jun 05 2018 13:56
science is very fragmented
Félix C. Morency
@fmorency
Jun 05 2018 13:58
there are pros and cons imo
Paolo Di Tommaso
@pditommaso
Jun 05 2018 13:58
yes, definitely
Shellfishgene
@Shellfishgene
Jun 05 2018 14:41
@pditommaso You were right of course, it was a conda problem, it did for some reason not update jdk to the required version for fastqc. I'm turning it into a nextflow problem though ;). So now I have a conda env for fastqc, this works in a bash script submitted to the cluster. However if I use the brand new conda directive to load that env, it does not.
Paolo Di Tommaso
@pditommaso
Jun 05 2018 14:41
However if I use the brand new conda directive to load that env, it does not.
how do you specify the env ?
Shellfishgene
@Shellfishgene
Jun 05 2018 14:42
conda '/home/user/miniconda2/envs/fastqc'
in the process for fastqc
It seems to work, I have source activate /sfs/fs6/home-geomar/smomw240/miniconda2/envs/fastqc in my .command.run
Paolo Di Tommaso
@pditommaso
Jun 05 2018 14:45
so what's the problem ?
Shellfishgene
@Shellfishgene
Jun 05 2018 14:46
I get the java fonts problem again.
Paolo Di Tommaso
@pditommaso
Jun 05 2018 14:46
LOL
goto :point_up: June 5, 2018 2:31 PM
Shellfishgene
@Shellfishgene
Jun 05 2018 14:47
Well, now I only get it with the nextflow .command.run, when I use my bash script it works...
Shellfishgene
@Shellfishgene
Jun 05 2018 14:59
@pditommaso I found the problem: As I had not used conda envs before, I had set the nf config file to add the conda/bin dir to PATH. Thus nextflow changed PATH in the .command.run after setting the conda env. I guess that changed the java from the env back to the old one.
Maybe nf could check for changes to PATH when setting conda environments.
Paolo Di Tommaso
@pditommaso
Jun 05 2018 15:02
you are saying the the java in your custom PATH had priority over the one in the conda env ?
Shellfishgene
@Shellfishgene
Jun 05 2018 15:02
It seems that way, I just removed the custom PATH from the nf config and now it works correctly.
Paolo Di Tommaso
@pditommaso
Jun 05 2018 15:03
well, that's a limit in the thing NF can manage
Shellfishgene
@Shellfishgene
Jun 05 2018 15:04
Would it help to just change the position of env variable setting and conda env loading in the .command.run file? Not sure how this conda stuff works, as you can probably tell...
Paolo Di Tommaso
@pditommaso
Jun 05 2018 15:07
not sure.
Luca Cozzuto
@lucacozzuto
Jun 05 2018 15:53
Hi @pditommaso , using the lib folder in some way damaged the caching
WARN: [QConRawReads (sim_read2.fastq.gz)] Unable to resume cached task -- See log file for details
[3f/60e020] Cached process > getReadLength
WARN: [trimReads (sim)] Unable to resume cached task -- See log file for details
WARN: [trimReads (test)] Unable to resume cached task -- See log file for details
WARN: [QConRawReads (sim_read1.fastq.gz)] Unable to resume cached task -- See log file for details
WARN: [QConRawReads (test_read2.fastq.gz)] Unable to resume cached task -- See log file for details
WARN: [QConRawReads (test_read1.fastq.gz)] Unable to resume cached task -- See log file for details
Paolo Di Tommaso
@pditommaso
Jun 05 2018 15:54
you damage the caching .. :joy:
Luca Cozzuto
@lucacozzuto
Jun 05 2018 15:54
Jun-05 17:52:06.883 [Actor Thread 4] WARN  nextflow.processor.TaskProcessor - [QConRawReads (sim_read2.fastq.gz)] Unable to resume cached task -- See log file for d
etails
com.esotericsoftware.kryo.KryoException: Unable to find class: QC
    at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
    at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
Paolo Di Tommaso
@pditommaso
Jun 05 2018 15:55
let me guess, you are declaring a QC instance in the script ..
Luca Cozzuto
@lucacozzuto
Jun 05 2018 15:55
yes
Paolo Di Tommaso
@pditommaso
Jun 05 2018 15:55
do that with def x = new QC(..)
Luca Cozzuto
@lucacozzuto
Jun 05 2018 15:59
let me guess in every process the name of the object has to be different?
I cannot use the same name in different processes?
Paolo Di Tommaso
@pditommaso
Jun 05 2018 15:59
no
Luca Cozzuto
@lucacozzuto
Jun 05 2018 15:59
because now I have this
ERROR ~ Variable `qualimp_memory` already defined in the process scope @ line 330, column 139.
   d-specific-reverse", memory:"${qualimp_m
                                 ^

_nf_script_e12cb6d0: 330: Variable `mode` already defined in the process scope @ line 330, column 164.
   ry:"${qualimp_memory}G", mode:mode, extr
                                 ^

2 errors
Paolo Di Tommaso
@pditommaso
Jun 05 2018 16:00
wat?
paste the code
Luca Cozzuto
@lucacozzuto
Jun 05 2018 16:02
process qualimap {
    tag { pair_id }

    input:
    file annotation_file

    set pair_id, file(bamfile) from STARmappedBam_for_qualimap

    output:
    set pair_id, file("QUALIMAP_${pair_id}") into QualiMap_for_multiQC 

    script:
    qualimp_memory = Math.round(0.6*task.memory.giga )
    mode = (params.single != "NO") ? 'pe' : 'se'  

    def qc = new QC(input:bamfile, annotation_file:annotation_file, output:"QUALIMAP_${pair_id}", strand:"strand-specific-reverse", memory:"${qualimp_memory}G", mode:mode, extrapars:"-s")
     qc.qualimapRNAseq()  
 }
Paolo Di Tommaso
@pditommaso
Jun 05 2018 16:03
weird
use
    def qualimp_memory = Math.round(0.6*task.memory.giga )
    def mode = (params.single != "NO") ? 'pe' : 'se'
Luca Cozzuto
@lucacozzuto
Jun 05 2018 16:04
yes
I should force myself to use def always before
Paolo Di Tommaso
@pditommaso
Jun 05 2018 16:04
this is a good thing ..
Luca Cozzuto
@lucacozzuto
Jun 05 2018 16:05
and now it works also the caching
well, I learnt how to kill the caching... :)
Paolo Di Tommaso
@pditommaso
Jun 05 2018 16:06
:grimacing:
Luca Cozzuto
@lucacozzuto
Jun 05 2018 16:18
However thanks again
:coffee: +2
Paolo Di Tommaso
@pditommaso
Jun 05 2018 16:18
:+1:
David Mas-Ponte
@davidmasp
Jun 05 2018 20:46
Hi! I have a very stupid question, sorry if this is not the place.
What is the recommended way to remove files used in all previous runs? Sometimes I do rm -rf work now I see there is the nextflow clean command. Do they do the same thing?
BTW, congrats for the release! The conda support will make my life much easier.