Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 25 16:07
    pditommaso closed #3435
  • Nov 25 16:07

    pditommaso on master

    Add support for custom conda ch… (compare)

  • Nov 25 16:07

    pditommaso on conda-custom-channels

    (compare)

  • Nov 25 16:01
    pditommaso synchronize #3435
  • Nov 25 16:00

    pditommaso on conda-custom-channels

    Fix conda defualt channels orde… (compare)

  • Nov 25 14:49
    jordeu opened #3438
  • Nov 25 14:45

    jordeu on tower_archive_basedir_fix

    TowerArchiver resolve envar pat… (compare)

  • Nov 25 13:49
    keiranmraine opened #3437
  • Nov 25 12:49

    pditommaso on master

    Add time directive to AWS Batch… (compare)

  • Nov 25 12:49
    pditommaso closed #3436
  • Nov 25 12:47
    llewellyn-sl opened #3436
  • Nov 25 11:55
    amizeranschi closed #3434
  • Nov 25 11:55
    amizeranschi commented #3434
  • Nov 25 11:54
    amizeranschi closed #3433
  • Nov 25 11:54
    amizeranschi commented #3433
  • Nov 25 11:38
    pditommaso commented #3434
  • Nov 25 11:27
    pditommaso review_requested #3435
  • Nov 25 11:27
    pditommaso opened #3435
  • Nov 25 11:23

    pditommaso on conda-custom-channels

    Add support for custom conda ch… (compare)

  • Nov 25 10:49
    nvnieuwk commented #3370
Riccardo Giannico
@giannicorik_twitter
I printed workflow.workDir in this line log.info "moving folder ${workflow.workDir} into ${workflow.workDir}_renamed " and it works correctly
Paolo Di Tommaso
@pditommaso
you are right, it should be
#!/bin/bash
set -e 
nextflow "$@"
mv work/ somewhere/
:)
if execution fail exit status == 1, then mv is not executed
Riccardo Giannico
@giannicorik_twitter
I need the opposite! I need mv to be executed if nextflow fails!
However I get it, thanks
nextflow "$@" || mv work/ somewhere/
Riccardo Giannico
@giannicorik_twitter
For the record.. I was trying a .nf solution instead of a bash one just because I trusted more the workflow.success instead of the exits status to determine if the pipeline fails for some reason. But if you say to trust the exist status I'll do!
Paolo Di Tommaso
@pditommaso
uh, trust is a big workd :)
Riccardo Giannico
@giannicorik_twitter
well from the code you linked seems like
if workflow.success is false , you always have exit status==1
I do not see any particular case where it should be different :P So I can use exit status
evanbiederstedt
@evanbiederstedt
@pditommaso :)
RE: "WARN: The channelcreatemethod is deprecated -- it will be removed in a future release"
What I meant to ask is whether there is any more information for whether/when we should edit our current workflows :)
*is there more information, not "is this more information"
Paolo Di Tommaso
@pditommaso
that's a work in progress, in principle it should not be needed
Stijn van Dongen
@micans

@pditommaso is Channel.create() going away? I am just looking at a pipeline where all the channel manipulations are listed before all the processes, e.g.

SAMPLED_FQ = Channel.create()
SAMPLED_FQ.map({ .. })
          .groupTuple().into({STAR_FQ; CR_FQ})
STAR_INPUTS = STAR_FQ.combine(STARIDX.mix(WHITELIST).collect())
// all process definitions below

this depends on Channel.create() I assume for this to work; (1) is that correct, (2) is Channel.create() going away, (3) if so can the above logic still be used?

Paolo Di Tommaso
@pditommaso
SAMPLED_FQ = Channel.create()
SAMPLED_FQ.map({ .. })
?
what this is doing?
Stijn van Dongen
@micans
the .. was removed, sorry
that's a long bit of code I did not copy
Paolo Di Tommaso
@pditommaso
the map is not the prob
what is supposed to be the source of SAMPLED_FQ
Stijn van Dongen
@micans
ok this is not my pipeline but one I am studying and have not used yet. I like how all its channel logic is before all the process logic. Pipeline is not open to sharing yet.
Paolo Di Tommaso
@pditommaso
to me the only use for create that makes sense is when used with choice
but I've seen so abuse that ideally it should it away
Stijn van Dongen
@micans
I'll make a toy example that we can discuss, OK?
something that works :-)
Paolo Di Tommaso
@pditommaso
ok ;)
Stijn van Dongen
@micans
ah, I just checked, we have channel.create() in our rnaseq, indeed with choice. Anyway, will be back later!
Stijn van Dongen
@micans

Allright, I'm back @pditommaso . The example is not perfect, but will do. It's the "optional process step" solved using tap and until (e.g. either run A-C or A-B-C). Using Channel.create() it's possible to put all the channel logic in one place:

#!/usr/bin/env nextflow

// This is a variant on http://nextflow-io.github.io/patterns/index.html#_problem_19
// using `tap` and `until` instead of Channel.empty().

params.includeB = false

ch_input = Channel.create()
ch_BC = Channel.create()

ch_input.flatMap().map { f -> [f.text.trim(), f] }.view()
  .tap { ch_AC }
  .until { !params.includeB }
  .set { ch_AB }

ch_AC.until { params.includeB }.mix(ch_BC).set{ ch_C }


process processA {        // Create a bunch of files, each with its ow sample ID.
  output: file('*.txt') into ch_input
  script: 'for i in {1..7}; do echo "sample_$i" > f$i.txt; done'
}

process processB {
  input:  set val(sampleid), file(thefile) from ch_AB
  output: set val(sampleid), file('out.txt') into ch_BC
  script: "(echo 'B process'; cat $thefile; md5sum $thefile) > out.txt"
}

process processC {
  publishDir "results"
  input: set val(sampleid), file(a) from ch_C.view()
  output: file('*.out')
  script: "(echo 'C process'; cat $a) > ${sampleid}.out"
}

It's not a great example as in this case the logic is quite convoluted, but I needed something small. Anyway, it seems a very useful thing to do, to list all the channel logic in one place; it fits nicely in the declarative paradigm.

Stijn van Dongen
@micans
Discussing this at work now ... different opinions about how useful it is. I like it though, perhaps a use case to bear in mind.
Paolo Di Tommaso
@pditommaso
I see, but ch_input = Channel.create() is redundant
ah wait
what prevents to declare
ch_input.flatMap().map { f -> [f.text.trim(), f] }.view()
  .tap { ch_AC }
  .until { !params.includeB }
  .set { ch_AB }

ch_AC.until { params.includeB }.mix(ch_BC).set{ ch_C }
after processA
then you won't need create any more or I'm missing something
Stijn van Dongen
@micans
@pditommaso yes that's possible; but I was excited by the possibility of keeping all the channel logic together; e.g. when you have 6 or more processes, to fully separate channels and processes in the file. There are pros and cons to that; when reading the source code one has to jump to go from process to channel (loss of locality), but there is no jumping from channel to channel (gain of locality). I like the clarity of channels together like that.
Paolo Di Tommaso
@pditommaso
with the new syntax a process will be invoked as a function, therefore there would not be anymore this need :sunglasses:
Stijn van Dongen
@micans
I was afraid you'd say this :-)
Paolo Di Tommaso
@pditommaso
ahah
Stijn van Dongen
@micans
Will the old syntax still be supported? Is it possible to have your script without an extra abstraction step?
Paolo Di Tommaso
@pditommaso
yes
Stijn van Dongen
@micans
:joy:
Well, the above is a point in Channel.create()'s favour. People (I'm one) will always find a way to write ugly buggy code, no matter what you do @pditommaso !
Paolo Di Tommaso
@pditommaso
:joy:
Riccardo Giannico
@giannicorik_twitter
sorry @micans maybe I do not understand everything on your example, but, why don't you use a logic like this?
process A {
   output: set val(sampleid), file('*.txt') into ch_Aout
   script: // something
}

if params.includeB {
   process B {
      input: set val(sampleid), file(thefile) from ch_Aout
      output: set val(sampleid), file('out.txt') into ch_Cin
      script: // something
   }
} else {
   ch_Cin= c_Aout
}

process C {
   input: set val(sampleid), file(thefile) from ch_Cin
   script: // something
}
Stijn van Dongen
@micans
@giannicorik_twitter the example was just to illustrate a different point. About your version, I don't like conditional code; it means the execution diagram is different depending on your parameters and I find it in general harder to read. But in some ways your example is more readable. I wonder if @pditommaso has more insights into conditional code?
Riccardo Giannico
@giannicorik_twitter
@micans , ah ok, so it's just a matter of taste :)
I personally prefer to turn on and off my optional steps using parameters and saving the command line I used (I even print it out in the nextflow stdout for that reason) . So If I am sure what happens exactly inside each step.