These are chat archives for nextflow-io/nextflow

22nd
Oct 2018
Luca Cozzuto
@lucacozzuto
Oct 22 2018 08:34
Hi @micans, yes my solution was to try to solve this at script level. But I was wondering if there is a way to solve this also at NF level. Many thanks!
Paolo Di Tommaso
@pditommaso
Oct 22 2018 08:38
when the task fail => no output is generated => no downsteam task is executed
micans
@micans
Oct 22 2018 08:48
Yep makes sense.
Paolo Di Tommaso
@pditommaso
Oct 22 2018 08:49
but ?
:)
micans
@micans
Oct 22 2018 08:50
Am I missing a better idiom for the code below?
    star_aligned
        .choice(ch_star_accept, ch_star_reject)
            { namelogsbams -> check_log(namelogsbams[1]) ? 0 : 1 }

    ch_star_accept
    .map    { name, logs, bams -> [name, bams] }
    .into   { ch_featurecounts; ch_indexbam }

    ch_star_reject
    .map    { it -> "${it[0]}\tSTAR\tlowmapping\n" }
    .mix(ch_lostcause_irods)
    .set    { ch_lostcause }
There is a check on alignment rate in check_logs (we gratefully copied the rna-seq pipeline from nf-core long ago, have made many changes since), and then I want to keep track in a table which samples are being lost. Samples can also be lost for other reasons, e.g. in the 'irods' stage.
Luca Cozzuto
@lucacozzuto
Oct 22 2018 08:52
@pditommaso I know. I'm just wondering if there is a way to output a empty channel in case of failure
Paolo Di Tommaso
@pditommaso
Oct 22 2018 08:54
if the task is never executed you can use ifEmpty
@micans code looks good, ideally we would need a way to define a special output for error conditions
micans
@micans
Oct 22 2018 09:09
Thanks @pditommaso that's a relief. Utilising some of the cool features in NF. Yes, it would be great if there was a way to have error conditions. Perhaps a feature where there are different error sinks (in NF those would be channels)? Then I'd like to declare/use an error channel tied to samples, and could send my messages to that channel when I lose a sample. Is this worth opening an issue?
Paolo Di Tommaso
@pditommaso
Oct 22 2018 09:10
Yes, let's open a thread and discuss there
micans
@micans
Oct 22 2018 09:10
:+1:
micans
@micans
Oct 22 2018 09:27
Just thinking out loud still .. what I envision so far is the same I think as log.info plus an additional argument (denoting error sink/channel), where the combined output can be published. Perhaps you are thinking of something more general @pditommaso ?
Paolo Di Tommaso
@pditommaso
Oct 22 2018 09:29
the rationale should be to trigger an event if a process fails, I was thinking more
process foo {
  input:
  output:
  onError: 
  // do something
}
micans
@micans
Oct 22 2018 09:38
I see. That looks good. It should have the exit code available. What I envisioned is orthogonal to this; also I may want to log checkpoint events as described above, and those would not necessarily fit in the error paradigm, do you agree? As far as I can see there is A) onError feature and B) ability to log and publish logs as part of pipeline output.
Paolo Di Tommaso
@pditommaso
Oct 22 2018 09:42
it looks to me you are just trying to get a log file, while just using the default trace feature?
micans
@micans
Oct 22 2018 09:45
Well, I want a nice table, for now sampleid procname description but other applications may have different needs; also I would like to get this table into the multiqc process and output. I really want to be able to process this meta-data in the pipeline itself and present customised outputs to the user.
So it's not an error feature I want, more a 'logging in a channel' feature. Perhaps the way I've done it above is not easily compressable, if I want the same flexibility and expressiveness.
Paolo Di Tommaso
@pditommaso
Oct 22 2018 09:57
more than logging channel I would like to reasoning more on metadata tracking, we should have a chat with @emi80 in bcn
micans
@micans
Oct 22 2018 10:07
well meta-data tracking may cover what I want as well; I assume it does given that the example I gave above triggered this conversation so .... it's simply that metadata tracking is a better description than calling it logging? A big part of what I want is to be able to use this information in the pipeline itself and route it to processes, does that fit your vision? Looking forward to chat in real-life!
Paolo Di Tommaso
@pditommaso
Oct 22 2018 10:09
yes, above all more generic, this could allow the generation of a log file or also to trigger tasks executions
micans
@micans
Oct 22 2018 10:11
Sounds perfect. I can open an issue, but is this two issues (metadatatracking and onError) or just one?
Sounds like two ...
Paolo Di Tommaso
@pditommaso
Oct 22 2018 10:11
let call it execution metadata tracking and routing
micans
@micans
Oct 22 2018 10:12
Thunderbirds are go
Paolo Di Tommaso
@pditommaso
Oct 22 2018 10:12
ahah
Yasset Perez-Riverol
@ypriverol
Oct 22 2018 11:27
Hi everyone, do we have a simple way to pass a parameter to a workflow in the current nextflow version
?
Paolo Di Tommaso
@pditommaso
Oct 22 2018 11:30
like this ?
Yasset Perez-Riverol
@ypriverol
Oct 22 2018 12:58
thanks !!!!
Paolo Di Tommaso
@pditommaso
Oct 22 2018 12:59
it was easy :)
micans
@micans
Oct 22 2018 15:43

@paolo if I can pick your brain for a moment ... I have trouble making progress with a bug. The symptom is that multiqc fails on k8s. The error messag is docker not found, and in the .command.run file we have

(
docker run -i --memory 4096m -e "NXF_DEBUG=${NXF_DEBUG:=0}" -v /mnt/gluster/svd:/mnt/gluster/svd -v "$PWD":"$PWD" -w "$PWD" --entrypoint /bin/bash --name $NXF_BOXID quay.io/biocontainers/multiqc:1.6--py35h24bf2e0_0 -c "eval $(nxf_taskenv); /bin/bash /mnt/gluster/svd/tic-test/work/24/0d4095a6aba59a30f87d261f94d824/.command.stub"
) >$cout 2>$cerr &

But in all other .command.run files we see simply (for example)

(
/bin/bash /mnt/gluster/svd/tic-test/work/88/5245cd3bc3749ec5018aa9ba01e766/.command.stub
) >$cout 2>$cerr &

Everything seems to be similar otherwise, e.g. in terms of container specification in the config file. Any idea where to look next for clues?