Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 13 20:12
    haqle314 edited #1425
  • Dec 13 18:37
    kemin711 opened #1426
  • Dec 13 17:32
    bobamess commented #1414
  • Dec 13 17:23
    tverbeiren commented #1424
  • Dec 13 17:22
    tverbeiren commented #1424
  • Dec 13 17:21
    haqle314 opened #1425
  • Dec 13 15:44
    JonathanCSmith commented #1424
  • Dec 13 15:25
    tverbeiren commented #1424
  • Dec 13 14:39
    tverbeiren commented #1249
  • Dec 13 11:34
    bobamess commented #1414
  • Dec 13 11:00
    JonathanCSmith opened #1424
  • Dec 13 10:20
    pditommaso commented #1409
  • Dec 12 23:40
    haqle314 commented #1422
  • Dec 12 19:08
    emorice commented #1419
  • Dec 12 18:50
    brgew opened #1423
  • Dec 12 16:52
    micans opened #1422
  • Dec 12 15:25
    daudn edited #1421
  • Dec 12 15:24
    daudn edited #1421
  • Dec 12 15:21
    daudn opened #1421
  • Dec 12 15:15
    emorice synchronize #1419
Paolo Di Tommaso
@pditommaso
always travis ?
Phil Ewels
@ewels
hmm?
ah, didn't work as workflow.onComplete doesn't fire with this error
Paolo Di Tommaso
@pditommaso
a bad day ..
Phil Ewels
@ewels
oh no it did, it's just that didn't work either
hah, yup ;) Ok, git reset and let's just stick with writableInputMounts = true so that I can go home :D
Thanks everyone for debugging help! Sorry for spamming everyone.
Let me know if you have any ideas or if there's anything I can do to help debugging work..
Paolo Di Tommaso
@pditommaso
I'm trying to find a patch
Paolo Di Tommaso
@pditommaso
@mikheyev Sorry I saw only now your question
However, is there any way to gracefully recover from such an error and ideally relaunch the job?
yes, you can set errorStrategy = 'retry' to re-submit a failed job
Paolo Di Tommaso
@pditommaso
@ewels I've uploaded 0.25.1-SNAPSHOT that should solve the docker mount problem, you may want to give a try
Phil Ewels
@ewels
ok great! Commit here: ewels/NGI-MethylSeq@c1136c8
Phil Ewels
@ewels
Two jobs here - I specified the snapshot for one (the one that passed) but not the other (the one that failed again)
Paolo Di Tommaso
@pditommaso
nice, thanks!
Alexander Mikheyev
@mikheyev
@pditommaso Weirdly, the jobs just disappear without an error being generated. I am working with the cluster admins to figure out the reason for it. I am not sure what Nextflow's behavior should be in such a pathological case.
Nexflow actually tries to kill some of these nonexisting jobs, but can't
```Jun-28 11:43:41.743 [main] DEBUG n.executor.AbstractGridExecutor - Unable to killing pending jobs
  • cmd executed: scancel 13931726 13931728 13931729 13931733 13931740 13931746 13931747 13931748 13931751 13931755 13931756 13931759 13931763 13931764 13931769
  • exit status : 130
  • output :
    scancel: error: slurm_kill_job2() failed Job/step already completing or completed
    scancel: error: slurm_kill_job2() failed Job/step already completing or completed
    scancel: error: slurm_kill_job2() failed Job/step already completing or completed
    scancel: error: slurm_kill_job2() failed Job/step already completing or completed
    scancel: error: slurm_kill_job2() failed Job/step already completing or completed
    ```
So, maybe the job poller can notice that a job is not there, and throw an error?
Alexander Mikheyev
@mikheyev

I am trying to split a series of paired-end libraries, and then phase them together, but some of the resulting files appear to be out of sync:

forward =  Channel
     .fromPath("data/reads/*R1*.fastq.gz")     
     .map{ file -> tuple(file.name.replaceAll(/_R1.*/,""), file) }
     .splitFastq( file: true, by: params.chunkSize)


reverse =  Channel
     .fromPath("data/reads/*R2*.fastq.gz")
     .map{ file -> tuple(file.name.replaceAll(/_R2.*/,""), file) }
     .splitFastq( file: true, by: params.chunkSize)

process map {

    input:
    file ref from file( params.ref )
    set file(forward), file(reverse) from forward.phase(reverse).map{f, r -> tuple(f[1], r[1])}

This seems to work the vast majority of the time, but once in a while the mapping process finds read files that are not properly paired. What am I doing wrong?

Alexander Mikheyev
@mikheyev
Actually, I think I know what's wrong -- the library name is not unique to the split files, but how do I make it so?
Paolo Di Tommaso
@pditommaso
I think the problem is that phase don't work as expected here
by doing that you will have many chunks with the same sample ID, thus they may be out of sync
Paolo Di Tommaso
@pditommaso
you can use something like the following
params.chunkSize = 10_000

Channel
  .fromFilePairs('data/reads/*_{1,2}.fq.gz', flat:true)
  .into { fwd; rev }

fwd = fwd.map { sample, file1, file2 -> tuple(sample,file1) }.splitFastq(file:true, by:params.chunkSize)  
rev = rev.map { sample, file1, file2 -> tuple(sample,file2) }.splitFastq(file:true, by:params.chunkSize)  

fwd.merge(rev) { left, right -> tuple(left[0], left[1], right[1]) }.println()
Said that, it think it would be handy that splitFastq would be able to split two or more files are at times. We were speaking with @skptic about that yesterday
Alexander Mikheyev
@mikheyev
Thanks!
Paolo Di Tommaso
@pditommaso
also, you should make sure that both read files contains the same number of reads (!)
Alexander Mikheyev
@mikheyev
I second the addition of a multiple file feature for splitFastq
Paolo Di Tommaso
@pditommaso
great
:)
regarding the problem with your cluster, thus what happens is that some job work directories are just dropped ?
Maxime Garcia
@MaxUlysse
HI @pditommaso quick question about singularity, Nextflow does not dowload images from shub yet, right?
Paolo Di Tommaso
@pditommaso
Not possible at this time, see #356
Maxime Garcia
@MaxUlysse
Oh yes, I remember reading this issue, I'll suscribe to it then
I'm trying to run CAW with singularity on one of UPPMAX cluster
It will probably be fun
;-)
Paolo Di Tommaso
@pditommaso
I guess so
Just download the image and use the file
Curious to know how it will perform
Maxime Garcia
@MaxUlysse
I'll keep you updated
Alexander Mikheyev
@mikheyev
@pditommaso For the cluster, it looks like some jobs launch, then suddenly disappear without an error log or any other trace that they existed
Paolo Di Tommaso
@pditommaso
!!
Alexander Mikheyev
@mikheyev
It is super-puzzling and appears to affect only members of my research group.
So, my guess it has something to do with lustre and the particular permission set I have.
Paolo Di Tommaso
@pditommaso
But the task work for created by NF are deleted as well?
*work directories
Alexander Mikheyev
@mikheyev
I don't think they are ever even created