These are chat archives for nextflow-io/nextflow

24th
Jan 2017
Eivind Gard Lund
@eivindgl
Jan 24 2017 09:29
process merge_all_vcfs {
  module 'BCFtools'
  input:
    file('vcf/*') from inputGzCh.toList()
    file('vcf/*') from inputGzIndexCh.toList()
  output:
    set val('all'), 'all.vcf' into mergedVcfCh
  """
  bcftools merge -o all.vcf -O v vcf/*.vcf.gz
  """
}
Caching is not working for this process, which is a problem for me since I do some computation at a later step. I remember reading about a similar problem caused by the toList() operation (I think) but I can't find that thread again and I can't remember the solution. Any pointers?
Paolo Di Tommaso
@pditommaso
Jan 24 2017 09:31
use toSortedList instead of toList
Eivind Gard Lund
@eivindgl
Jan 24 2017 09:39
@pditommaso Great, that makes sense! Thank you.
Mike Smoot
@mes5k
Jan 24 2017 20:29
Hi @pditommaso, is it possible to pass a comparator to the toSortedList() operator?
Paolo Di Tommaso
@pditommaso
Jan 24 2017 20:29
source code is best answer .. :)
let me check
yes
Mike Smoot
@mes5k
Jan 24 2017 20:32
awesome!
Mike Smoot
@mes5k
Jan 24 2017 20:41
Would you like me to add an example to the documentation for this?
Paolo Di Tommaso
@pditommaso
Jan 24 2017 20:41
yes please
Mike Smoot
@mes5k
Jan 24 2017 20:45
will do
Trevor Tanner
@tantrev
Jan 24 2017 22:07
Any advice for what Unix signal should be sent to NF to kill all of its pending tasks? I'm using NF to submit slurm jobs, but when I send NF SIGINT (through scancel), only NF quits, while leaving all of its children slurm jobs alive.
Paolo Di Tommaso
@pditommaso
Jan 24 2017 22:21
well, the expected behaviour is that pending slurm tasks are killed
may you share the .nextflow.log file ?
Trevor Tanner
@tantrev
Jan 24 2017 22:35
of course, I just uploaded it to http://epilog.bio/files/.nextflow.log
Paolo Di Tommaso
@pditommaso
Jan 24 2017 22:36
:
Jan-24 15:26:05.924 [SIGINT handler] DEBUG nextflow.Session - Session aborted -- Cause: SIGINT
Jan-24 15:26:05.927 [Thread-2] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm)
Jan-24 15:26:05.929 [Thread-1] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local)
Jan-24 15:26:05.929 [main] DEBUG nextflow.Session - Session await > all process finished
Jan-24 15:26:05.929 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jan-24 15:26:05.947 [main] WARN  n.processor.TaskPollingMonitor - Killing pending tasks (8)
Jan-24 15:26:05.955 [Actor Thread 4] DEBUG nextflow.processor.TaskProcessor - <checkCache> After stop
Jan-24 15:26:05.955 [Actor Thread 22] DEBUG nextflow.processor.TaskProcessor - <extractSRA> After stop
Jan-24 15:26:06.033 [main] WARN  n.processor.TaskPollingMonitor - Killing pending tasks (100)
:
it says killing pending tasks ..
Trevor Tanner
@tantrev
Jan 24 2017 22:37
Yeah, I noticed that's what is says. It seems to still leave jobs when I check with squeue though.
Paolo Di Tommaso
@pditommaso
Jan 24 2017 22:38
how many jobs were left
Trevor Tanner
@tantrev
Jan 24 2017 22:40
When I just checked, there were 15. But a lot had finished their task by that point as well - am redoing right now to get a better idea.
I just redid it, and there are 75 remaining.
Paolo Di Tommaso
@pditommaso
Jan 24 2017 22:42
and killing line in the log how many it says ?
Trevor Tanner
@tantrev
Jan 24 2017 22:44
100
Paolo Di Tommaso
@pditommaso
Jan 24 2017 22:44
weird
um .. you could try to enable fine logging to check if the scancel command is correctly executed
nextflow -trace nextflow.executor.AbstractGridExecutor run .. etc
Trevor Tanner
@tantrev
Jan 24 2017 22:48
awesome, thx - will try rn
Paolo Di Tommaso
@pditommaso
Jan 24 2017 22:49
you should find in the log one or more lines like TRACE Kill command: ....
Trevor Tanner
@tantrev
Jan 24 2017 22:57
ok so http://epilog.bio/files/.nextflow.log now has the latest with the fine logging
Paolo Di Tommaso
@pditommaso
Jan 24 2017 23:00
um, the scancel is there
it cancel 100 jobs but only 25 are actually killed
Trevor Tanner
@tantrev
Jan 24 2017 23:01
Is it possible that NF isn't waiting for the command to finish? right
Paolo Di Tommaso
@pditommaso
Jan 24 2017 23:01
um
no, it isn't
do you think this stops the jobs killing ?
Trevor Tanner
@tantrev
Jan 24 2017 23:02
lol, just throwing out ideas. you mean scancel? the command I'm using specifically is "scancel -s INT [jobid.jobstepid]"
Paolo Di Tommaso
@pditommaso
Jan 24 2017 23:03
-s INT should be the default ?
Trevor Tanner
@tantrev
Jan 24 2017 23:05
oh cool, didn't know. is there any way to customize the scancel behavior? perhaps it's just a network thing and the "--ctld" might fix it
Paolo Di Tommaso
@pditommaso
Jan 24 2017 23:06
unfortunately no, but not waiting the command to finish could be reason
I'm preparing a patch, so u can give a try
Trevor Tanner
@tantrev
Jan 24 2017 23:07
ur awesome, thank you
Paolo Di Tommaso
@pditommaso
Jan 24 2017 23:23
done
NXF_VER=0.23.2-SNAPSHOT nextflow run .. etc
Trevor Tanner
@tantrev
Jan 24 2017 23:26
b-e-a-utiful
that did the trick
Paolo Di Tommaso
@pditommaso
Jan 24 2017 23:27
:+1:
great work, congrats !
I can go to sleep happy now :)
Trevor Tanner
@tantrev
Jan 24 2017 23:29
This was all you, I appreciate it. :smile: I get to crunch numbers now haha
Paolo Di Tommaso
@pditommaso
Jan 24 2017 23:30
enjoy it ! ;)