These are chat archives for nextflow-io/nextflow

19th
Mar 2019
Pierre Lindenbaum
@lindenb
Mar 19 08:24
@pditommaso I was too fast yesterday about my cache problem. https://gitter.im/nextflow-io/nextflow?at=5c8f9ee70d719050572537dc I set cache lenient everywhere, an re-ran my workflow with now ~2000 genomic regions. At some point the workflow manager failed , I tried to re-submit using -resume but it started again from the first process callBcftools. . I also tested with cache=false (hoping that NF would just consider the return status=COMPLETED of the process, not the file itself(?)) . I posted my nf file here https://gist.github.com/lindenb/a0ca7ec35ae36c9d24f338190768c73c . I don't think the upstream process is not-deterministic (?).
Paolo Di Tommaso
@pditommaso
Mar 19 08:25
:joy:
Pierre Lindenbaum
@lindenb
Mar 19 08:25
Is there something in the log that would allow me to know why NF choose to re-run a process ?
Paolo Di Tommaso
@pditommaso
Mar 19 08:27
a process is (re)executed when the inputs+script give a new hash key
you can use the -dump-hashes and check those hashes, not so friendly tho
Luca Cozzuto
@lucacozzuto
Mar 19 08:30
the fastest way is to delete the temporary folder corresponding to that process
Matthew Pocock
@drdozer
Mar 19 10:33
are there example nextflow scripts for retrieving sequences by ID from swissprot, and running blast against nr? or do I have to download everything locally?
Yasset Perez-Riverol
@ypriverol
Mar 19 12:08
Hi, I have a tool that can take like 20 different parameters to generate different 20 outputs (figures). The tool needs to be execute one parameter at a time. I was planning to use a for loop to execute 20 times the same script (to avoid 20 channels) but I can find a nice example in github.
Maxime Garcia
@MaxUlysse
Mar 19 12:57
Paolo Di Tommaso
@pditommaso
Mar 19 12:57
it looks a bad idea ..
Maxime Garcia
@MaxUlysse
Mar 19 12:58
That's exactly how I started when I managed to make and endless loop
Yasset Perez-Riverol
@ypriverol
Mar 19 12:59
The problem is that the software has been design in that way, Im trying to work with the developer in a different way but it will take time. I try to do something like:
Paolo Di Tommaso
@pditommaso
Mar 19 12:59
use each
Yasset Perez-Riverol
@ypriverol
Mar 19 13:00
I will try each
Yasset Perez-Riverol
@ypriverol
Mar 19 13:24
each works !!! 👍
Paolo Di Tommaso
@pditommaso
Mar 19 13:24
well done
it seems still a lot to learn about NF :wink:
Yasset Perez-Riverol
@ypriverol
Mar 19 13:27
Yes!!! also it seems like a lot of room for examples and github WFs :)
Paolo Di Tommaso
@pditommaso
Mar 19 13:27
great
Tobias "Tobi" Schraink
@tobsecret
Mar 19 17:11
@all The FAQ is due for some renewal. What are some common questions/ confusions you have come across in the gitter/ your own work, that you think would be beneficial to have in the FAQ? This can include things that are in the official documentation but that you would have searched for with a different question.
One example that I would like to add:
Q: Nextflow crashes when run on the cluster, what's wrong? Why is Nextflow using so much memory?
A: Nextflow by default does not have a limit to how much memory it will use. On a need-basis Nextflow will thus use increasing amounts of memory. One can mitigate this by specifying the minimum (-Xms) and minimum (-Xmx) amounts of memory Nextflow gets to use like so: NXF_OPTS='-Xms512m -Xmx2G' nextflow run [etc] Nextflow should run just fine with these settings, it does not actually strictly require all the RAM.
Paolo Di Tommaso
@pditommaso
Mar 19 18:23
good point
I would says that NF being a JVM based application uses the Java default memory allocation that tends to be quite eager
1/3 of available mem if I remember well, therefore user should use the NXF_OPTS .. etc
Laurence E. Bernstein
@lebernstein
Mar 19 19:38

@tobsecret There are a few I have come across in my recent work starting to use Nextflow. Definitely some of the questions on the google group would be good. I can't tell you how much time I spent on these two:

Google: "Getting an output value from a python script within nextflow process"
Q: How do you get a value from a script into another process
A: Write the value to a file and then use that file as a value channel to the next process. Alternatively do this with stdout.

Google: Parsing list of IDs with splitText (get rid of new line character)
Q: How do I get rid of new line character when using splitText()
A: Instead use splitCSV() or do the following:
inputChannel.splitText().map{it -> it.trim()}

Tobias "Tobi" Schraink
@tobsecret
Mar 19 19:55
@lebernstein Ooooh, I like those!
Tobias "Tobi" Schraink
@tobsecret
Mar 19 20:02
This message was deleted
Stephen Ficklin
@spficklin
Mar 19 21:31
Hopefully this is a quick question.... I'm using a shell script for a process in a workflow. I have some if statements checking certain error conditions so that I can hand those cases in different processes. The program I'm calling in the bash code has a return code that I analyze to deal with issues as I don't want the workflow to stop. But I need to set the error code that Nextflow will see. I'm using the exit command to set an error code but Nextflow seems to always get an error code of 1 and stops even though I've provided a different one.
Is it possible to use the bash exit command inside of a process shell script to force a specific exit code?
Laurence E. Bernstein
@lebernstein
Mar 19 23:03
@spficklin Will validExitStatus directive help you?