These are chat archives for nextflow-io/nextflow

20th
May 2019
Rad Suchecki
@rsuchecki
May 20 01:41

@evanbiederstedt @stevekm @rsuchecki For my purposes, the html report would probably be difficult to use since it would require a lot of parsing. My Nextflow workflow is part of a larger process and my goal was to provide a simple file for post-processing that could be used by follow on processing to tell which samples were processed successfully and which were not so that appropriate actions could be taken (possibly automatically).
In the end my workflow generates a JSON file that looks like:

{
  "Status" : "Failure",
  "Sample_1" : "Successful",
  "Sample_2" : "Successful",
  "Sample_3" : "Failed",
}

The report COULD then be used to track down the problem, but I also have my status files (1 per sample) that are plain text and can be read to see everything you need to know (hopefully) about the issue.

There seems to be quite a bit of overlap here with the weblog feature in NF https://www.nextflow.io/docs/latest/tracing.html#weblog-via-http
Wonder if there should be an option to send final weblog POST payload for each task into a file.

evanbiederstedt
@evanbiederstedt
May 20 03:38

Wonder if there should be an option to send final weblog POST payload for each task into a file.

@rsuchecki Sorry, I don't quite follow you. Isn't this what the weblog is?

Rad Suchecki
@rsuchecki
May 20 03:42
Well you need a webserver to listen and capture all that right? I was thinking of using the same JSON content but having it written directly to a local file - without the web bit - just the log ;-)
I already use this kind of approach to capture general workflow metadata but not for individual tasks. @evanbiederstedt
evanbiederstedt
@evanbiederstedt
May 20 04:58

having it written directly to a local file - without the web bit - just the log ;-)

Ah yes! This does make a great deal of sense, and would solve quite a few problems I think. Such a feature would be invaluable, particularly for individual tasks and individual inputs.

Shall we organize a PR? :) @rsuchecki

Rad Suchecki
@rsuchecki
May 20 05:21
Possibly, need to get a discussion going. Having the ability to easily dump metadata JSON is one thing but there should be quite a bit of overlap with #903. It seems to me #1139 is relevant as well.
Mazen Mahdi
@Shamanga_13_twitter
May 20 10:25
Hello,
How can we initialize a channel only after a certain process is finished?
Currently if we have process_1 and process_2 that are piped to each other with a mock dependency, then if in process_2 i created a channel to read some folder names that were created by process_1, this channel is empty, because this folder was empty when the pipeline was initialized.
I know this can be solved by piping the output folder from process_1 to process_2 instead of applying a mock dependency, but I was just curious
Luca Cozzuto
@lucacozzuto
May 20 12:28
Hi all,
is there a way to try a process 3 times and in case it fails to just skip it?
micans
@micans
May 20 12:31
This is possible I would think .. hold on.
We have this e.g. as a config setting:
errorStrategy = { task.exitStatus == 130 && task.attempt <= process.maxRetries ? 'retry' : 'ignore' }
Luca Cozzuto
@lucacozzuto
May 20 12:33
wonderful
:-D Basically the same as @micans does ;-)
Luca Cozzuto
@lucacozzuto
May 20 12:33
@pditommaso please let's add this to the documentation, it is a real plus!
thanks to both!
micans
@micans
May 20 12:34
:+1: @apeltzer and @lucacozzuto ... there is something nagging me that I have a todo item because I don't fully understand the retry regime ... but never mind :-)
Alexander Peltzer
@apeltzer
May 20 12:42

@pditommaso please let's add this to the documentation, it is a real plus!

I agree that would be nice :+1:

Paolo Di Tommaso
@pditommaso
May 20 12:43
oh yes, PR !
Alexander Peltzer
@apeltzer
May 20 12:43
Ok :-D
Paolo Di Tommaso
@pditommaso
May 20 12:44
:D
Luca Cozzuto
@lucacozzuto
May 20 12:46
ehm I get an error :)
    withLabel: little_comp {
        maxForks = 1
        maxRetries = 3
        errorStrategy = { task.attempt < process.maxRetries ? 'retry' : 'ignore' }
    }
is there any problem with this code?
maybe I should remove process.
Luca Cozzuto
@lucacozzuto
May 20 12:52
this is the error
java.lang.IllegalArgumentException: Cannot compare java.lang.Integer with value '1' and groovy.util.ConfigObject with value '[:]'
Paolo Di Tommaso
@pditommaso
May 20 12:57
all config?
Luca Cozzuto
@lucacozzuto
May 20 12:58
si
I solved with this
    withLabel: little_comp {
        maxForks = 1
        maxRetries = 3
        errorStrategy = { task.attempt <  3 ? 'retry' : 'ignore' }
    }
Paolo Di Tommaso
@pditommaso
May 20 12:58
well done
Luca Cozzuto
@lucacozzuto
May 20 13:00
this was really really useful. Many thanks!
hydriniumh2
@hydriniumh2
May 20 17:29
Hi, is there any way to run a nextflow process with a docker executable rather than inside the docker?
Laurence E. Bernstein
@lebernstein
May 20 17:39
@hydriniumh2 If I understand you correctly (using a container with ENTRYPOINT), I had the same question a while back. Previously there was a way to do this but it was deprecated in the latest release. I asked Paolo about it and I don't recall his exact response but he told me there is no intention to support it.
hydriniumh2
@hydriniumh2
May 20 17:40
Ah bummer, thats good to know, thanks!
evanbiederstedt
@evanbiederstedt
May 20 18:22

Possibly, need to get a discussion going. Having the ability to easily dump metadata JSON is one thing but there should be quite a bit of overlap with #903. It seems to me #1139 is relevant as well.

@rsuchecki Yes, let's start organizing on this.

As for the original motivation of the discussion, it appears the best ideas so far are:
(in my opinion, there should be others)
--- filters in the channels to remove obviously poor QC files @stevekm
--- something like the following
errorStrategy = { task.exitStatus == 130 && task.attempt <= process.maxRetries ? 'retry' : 'ignore' }
--- writing the individual sample status into a single file, which could be parsed by the user for debugging, e.g. the ability to a dump metadata JSON/some other file for quick parsing of "failed" inputs

@apeltzer
https://github.com/nf-core/tools/blob/4ab80b83cd8b6562a88bb38b6aa0a3d082fd4de6/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/conf/base.config#L19
:-D Basically the same as @micans does ;-)

I'll try that as well, something like

errorStrategy = { task.exitStatus in [143,137,104,134,139] ? 'retry' : 'finish' }
maxRetries = 1
maxErrors = '-1'
Anthony Ferrari
@af8
May 20 20:30

I am trying to launch a nf pipe with singularity[3.1.1] and slurm[18.08.6-2] (Native programs + slurm is working fine).
When I launch the pipe with Singularity containers, I obtain the following error message :

  + NXF_ENTRY=nxf_trace
  .command.run: line 21: syntax error: unexpected redirection

Line 21 corresponds to this bit of code in .command.run

    while read P PP;do
        ALL_CHILDREN[$PP]+=" $P"
    done < <(ps -e -o pid= -o ppid=)

Any idea of what is happening ?