These are chat archives for nextflow-io/nextflow

20th
Nov 2018
Maxime HEBRARD
@mhebrard
Nov 20 2018 01:58
hello ... I have an unexpected behavior of nextflow ... all my tasks completed , but the workflow do not exit !

the html report and the trace show all task completed ... but at the end I have a command

chan.subscribe { println "${it}" }

workflow.onComplete { log.info "-END-" }

The print from the channel is working, but the onComplete is not fired ...

Rad Suchecki
@rsuchecki
Nov 20 2018 02:54
how do you create that channel @mhebrard? You may have to .close() it
Rad Suchecki
@rsuchecki
Nov 20 2018 03:03
@tbugfinder .flatMap()
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:03
hmm I have multiple channels ...but all are used
Rad Suchecki
@rsuchecki
Nov 20 2018 03:03
what about chan @mhebrard ?
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:07
ng.png
oh subscribe to not close the channel ?
as you see all my channels are input of some processes ... only the last one cCounted is not input, but has the subscribe ...
is there a way to know what channel I may need to close ?
Rad Suchecki
@rsuchecki
Nov 20 2018 03:10
does it finish if run without subscribe?
normally you don't have to close yourself
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:10
when I run the nextflow, subscribe work well
Rad Suchecki
@rsuchecki
Nov 20 2018 03:11
sorry, misunderstood your question
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:13
ok :) ... my problem is I run the flow (shown as DAG above) ... all my tasks completed (checked on the report and the trace) but the nextflow never end (workflow.onComplete() not fired) ... I donno what keep my workflow to complete
Rad Suchecki
@rsuchecki
Nov 20 2018 03:16
I'd check .nextflow.log
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:20
oh yes ... I have an entry in the logs :
no more task to compute -- The following nodes are still active: 
[process] pSorting
status=ACTIVE
port 0 :(queue) OPEN; channel: bam
and another process with similar info
now have to understand why my process stay active ... or why my channel stay open ^^"
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:26
is it a problem with mix directive ?
Rad Suchecki
@rsuchecki
Nov 20 2018 03:26
Nope. I think some input channels perhaps bam do not get closed, as for why, it may depend on how they are created
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:28
hmm ok ...
let me come back ...
(ok from here I can copy paste better lol)
here the process
process pSorting {
  tag "${bam[0]}"
  scratch '/tmp'
  cpus params.threads

  publishDir path: "${params.outstar}", mode: 'copy', pattern: '*.sorted.*', saveAs: {filename -> "${bam[0]}/$filename"}
  publishDir path: "${params.outstar}", mode: 'copy', pattern: '*Log*', saveAs: {filename -> "${bam[0]}/$filename"}

  input:
    val bam from cBams

  output:
    file '*Log*'
    file '*.sorted.bai'
    set val("${bam[0]}"), file('*.sorted.bam') into cSorted
the input bam come from
cBams = Channel.create()
  .mix(cBamCreated, cBamExists.map{
    [it.name, file("${params.outstar}/${it.name}/${it.name}_Aligned.out.bam")]
  });
Rad Suchecki
@rsuchecki
Nov 20 2018 03:34
  1. No need to create the channel explicitly here
    cBams = cBamCreated.mix(cBamExists.map{
     [it.name, file("${params.outstar}/${it.name}/${it.name}_Aligned.out.bam")]
    });
  2. Possibly same with cBamCreated judging by the name
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:34
cBamCreated is an output from another process
Rad Suchecki
@rsuchecki
Nov 20 2018 03:36
more generally if you use .create() explicitly I think you need to .close()explicitly
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:36
oh I see
Thanks for the help .. I will check that
hmm but then how do I manage a choice directive ...
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:45
// Create Channel
  cBaiExists = Channel.create()
  cBamExists = Channel.create()
  cBamNotExists = Channel.create()

  Channel.fromPath(params.samples)
    .splitCsv(header: ['id', 'name'], skip: 1, sep: '\t', strip: true)
    .choice(cBaiExists, cBamExists, cBamNotExists) {a -> file("${params.outstar}/${a.name}/${a.name}_Aligned.sorted.bai").exists() ? 0 : file("${params.outstar}/${a.name}/${a.name}_Aligned.out.bam").exists() ? 1 : 2}
that is the origin of my bam channels ... basicaly, I test if my Bam or Bam index already exists in my output directory, then I can choose if I need to create the bam or or create the index, or just processed the bam ...
Maxime HEBRARD
@mhebrard
Nov 20 2018 03:51
but yeah my 2 processes that stay active have a mix channel as input .. I try to modify the channel creation with your suggestion and see what happen @rsuchecki
Maxime HEBRARD
@mhebrard
Nov 20 2018 06:03
Apparently using the mix directive the proper way solved my issue of channel open ... thanks
tbugfinder
@tbugfinder
Nov 20 2018 07:20
@rsuchecki Thx!!
Maxime HEBRARD
@mhebrard
Nov 20 2018 07:26
Question: I process a queue of files through multiple processes. Once all my processes are done on all my files (i.e. after the output of my last process) I wish to run a "last step process" ... how do I articulate my channels?
anonymous
@naveenmeena584
Nov 20 2018 08:33
i have written some bunch of scripts to initiate pipeline working fine, now i want to use it with Nextflow , my doubt is should i have to rewrite all scripts from scratch or i can use these scripts with Nextflow, i need suggestion in that.how i can use nextflow for already developed pipeline in any programming language.
second question is can we run snakemake based workflow directly on Nextflow, what are changes one have to make for running successfully using Nextflow Workflow??
Evan Floden
@evanfloden
Nov 20 2018 10:05
@naveenmeena584 Sure you can use your existing scripts. Wrap the script code in a process where you define the script inputs and outputs.
Paolo Di Tommaso
@pditommaso
Nov 20 2018 14:30
it remained at prototype stage, we realised it's too complicated and it's not worth the effort
regarding the UI, I'm not aware of any public project unfortunately but there are several ongoing initiative, maybe soon something will be released
what is the way of converting CWL into Nextflow
there isn't
there are plenty of cwl runner, makes no sense for nextflow to be just anotehr one
you are welcome to create one for nextflow
naveen584
@naveen584
Nov 20 2018 14:36
what you thing it will be helpful for users who want to use nextflow with GUI
or it will be not going to worth the effort
micans
@micans
Nov 20 2018 14:39
Philosophically I consider the keyboard a GUI too, but I realise this is not a fully constructive contribution. More to the point, I always wonder who the users of such a GUI will be, and who they will e-mail when the GUI breaks, and how much work will have been gainer or lost.
Evan Floden
@evanfloden
Nov 20 2018 15:39
I generally feel the same as @micans but I can see the appeal for some people. For a GUI to work properly without the need for knowledge or interaction with the code, the pipeline must be REALLY well developed and documented. To be useable, it would have to throw correct errors, know the correct data types, maybe provide even hints etc. It is for this reason that the focus so far has been on first describing parameters of pipeline which can be seen here: nextflow-io/nextflow#866.
micans
@micans
Nov 20 2018 16:25
Nice thread in that issue, seems like the right way of going about it, as each incremental step will be useful. As for GUI to work properly, yes really well developed and documented code needed indeed, and it additionally needs a very robust back-end. Simple things like disk quota and permissions can throw up errors, and I'm sure there are a few more that I forget. I feel this can be useful for production pipelines, where both hardware and software are battle-tested and hardware and quotas are monitored by someone.