These are chat archives for nextflow-io/nextflow

3rd
May 2019
Anna Syme
@AnnaSyme
May 03 03:07
Can Nextflow print a short summary of results to screen at workflow completion?
Laurence E. Bernstein
@lebernstein
May 03 06:01
@AnnaSyme That can definitely be done. The method would depend on what you mean by "summary of results". What I do is have a last process that runs after completion that does this sort of thing. You can add an output val set to true in the last process which is an input to your summary process so that it will wait for completion of your last process. Then you can pass in any values you wish to print out. Otherwise you can use the onComplete() handler, but that can not access values from other processes unless you do a little finessing (I think).
This message was deleted
Anna Syme
@AnnaSyme
May 03 06:32
thanks @lebernstein . I've also found that I need echo true at the start of the process, and that adding the .collect() to the channel being used as the final input.
Adam Nunn
@bio15anu
May 03 08:50
Can anyone explain how does Nextflow inherit new file permissions when setting up the workDir? Even with global umask 0002, every time somebody calls my pipeline the directories are generated with drwxr-sr-x permissions, which means that multiple people cannot work from the same workDir. Pipelines are executing with SLURM and I'm not sure if maybe the problem is there or with Nextflow.
Chelsea Sawyer
@csawye01
May 03 10:05
@pditommaso will do, thanks!
Paolo Di Tommaso
@pditommaso
May 03 11:33
Quick reminder that registration for Nextflow Camp is open
Vladimir Kiselev
@wikiselev
May 03 13:16
is there a way of putting -with-report to the config file?
Paolo Di Tommaso
@pditommaso
May 03 13:17
report.enabled = true
Vladimir Kiselev
@wikiselev
May 03 13:17
Beautiful! :+1:
Laurence E. Bernstein
@lebernstein
May 03 16:18
Is there a way to get the return value of the script in a process in order to use it in the afterScript directive?
Paolo Di Tommaso
@pditommaso
May 03 16:18
I fear no
Cedric
@Puumanamana
May 03 19:15
I have a weird issue, I'm not sure how to troubleshoot it:
Some of my processes terminate without an error : the pipeline just stops and makes as if everything over. This behavior is really hard to reproduce, but happens more on big runs.
I tried to go to the scratch directory of a process that did not finish, and there is also nothing to use (nothing in the .command.err), the log and output file just show the beginning of the normal output (as if someone would have paused the process). Normally, nextflow is supposed to throw an error when the outputs are not there, but nothing here. Any idea?
Laurence E. Bernstein
@lebernstein
May 03 20:09
I have seen this happen and I think that it occurs when there is a problem in the process definition that is not an actual problem with the script. For instance, something goes wrong in the beforeScript or afterScript directive.
Stephen Kelly
@stevekm
May 03 20:34
any possibility of having Nextflow log to a database anytime soon?
I am getting to the point where I really, really need to be able to e.g. write the paths to the output files to some file, so that other programs can read from it
also in some kind of custom output format, too
like I need to record that Nextflow task make_signatures created file Sample1.signatures.json for sample Sample1 and record the path to the file, so that some other program can come along and look up Sample1 and be able to find the file, if it exists.
Stephen Kelly
@stevekm
May 03 20:41
for example, for my custom pipeline reports, some samples may or may not have had certain Nextflow tasks run on them, depending on attributes of the data, and the files that are produced may have slightly different names, if they exist at all. So my report program needs to have a way to know what the filename for a certain file for a certain task is supposed to look like so it can try to find that file and do things with it. Similar to what MultiQC does. Also need this functionality in order to build web apps that can look through Nextflow file output and identify specific files for specific samples.
Cedric
@Puumanamana
May 03 20:43
@lebernstein : But the process runs fine most of the time. And I don't have any beforeScript or afterScript directive. The problem occurs during the script execution
Stephen Kelly
@stevekm
May 03 20:47
@Puumanamana are you seeing indications that the .command.run script finished successfully? For example do you see the .exitcode file?
I have been having issues that are kind of similar, our best guess is that OOM Killer is detecting process threads that are using too much memory and killing them. Could be a process that something in your .command.sh is running, or it could be some child process of Nextflow that might be monitoring something for completetion, or possibly some combination thereof
Cedric
@Puumanamana
May 03 21:04
@stevekm : I indeed don't have the .exitcode file. The process is running a child process (in my case, I had the issue with both R or Mothur). It could be an OOM error, but usually I get a nextflow error when this happens. I have the output from the child process as well, but same, no error.
Another thing also, is that I can't resume the run after these issues, there is a lock file. If I remove it and resume, then it works just fine