These are chat archives for nextflow-io/nextflow

23rd
Nov 2016
Maxime Garcia
@MaxUlysse
Nov 23 2016 10:53
@pditommaso find some inconsistency when dealing with list as an input, made an issue #251
Paolo Di Tommaso
@pditommaso
Nov 23 2016 10:56
I will give it a look asap
Maxime Garcia
@MaxUlysse
Nov 23 2016 10:56
Thanks
Paolo Di Tommaso
@pditommaso
Nov 23 2016 12:31
@MaxUlysse Thanks for this nice blog post http://bioinfo-fr.net/nextflow-pour-votre-prochain-pipeline
Regarding the Strachan affair it's true that some years ago he wrote
I can honestly say if someone had shown me the Programming in Scala book by by Martin Odersky, Lex Spoon & Bill Venners back in 2003 I'd probably have never created Groovy.
But more recently he said that he doesn't use Scala anymore .. :)
;)
Maxime Garcia
@MaxUlysse
Nov 23 2016 12:45
@pditommaso Thanks a lot
Thank you about the Strachan affair, I didn't knew about it
Paolo Di Tommaso
@pditommaso
Nov 23 2016 12:48
yes, this story is frequently mentioned but is more a meme as you said
Maxime Garcia
@MaxUlysse
Nov 23 2016 12:50
I admit I didn't look more into it, and was just mentionning it
Paolo Di Tommaso
@pditommaso
Nov 23 2016 12:51
:ok_hand:
Matthieu Foll
@mfoll
Nov 23 2016 14:42
Hi @pditommaso, I have a quick question: I have multiple files in a set as an input of a process (https://github.com/IARCbioinfo/needlestack/blob/dev/needlestack.nf#L433):`file('TABLE/*')`. What I observe at the moment is that files are staged in the work dir without being renamed, while according to the documentation they should be (https://www.nextflow.io/docs/latest/process.html#multiple-input-files). The current behavior is what I wanted, but after re-reading the doc I’m not sure this is the intended behavior.
Paolo Di Tommaso
@pditommaso
Nov 23 2016 14:53
no because NF knows what u want :)
kidding apart I need to update the table with the * only
in that case it maintains the original file names
even if you are using a sub directory as in your case
Matthieu Foll
@mfoll
Nov 23 2016 14:55
Great, I just wanted to be sure this behaviour will stay like that
As nextflow-io/nextflow#243 being fixed broke most our pipelines for example ;-) (but I agree the new behavior makes more sense and it’s an easy fix)
Paolo Di Tommaso
@pditommaso
Nov 23 2016 14:59
sorry I should have announced that
Félix C. Morency
@fmorency
Nov 23 2016 15:02
broke our pipelines too :P
Paolo Di Tommaso
@pditommaso
Nov 23 2016 15:03
otherwise they would always work ! too boring !!
:)
Maxime Garcia
@MaxUlysse
Nov 23 2016 15:05
:+1:
Matthieu Foll
@mfoll
Nov 23 2016 15:24
And at least it justifies why we have some CircleCI tests running
Matthieu Foll
@mfoll
Nov 23 2016 15:32
first reaction was: CircleCI is broken (when you trust too much your pipelines and NF behind)
Paolo Di Tommaso
@pditommaso
Nov 23 2016 15:37
:)
Phil Ewels
@ewels
Nov 23 2016 16:04
hah, yes ours keep breaking too. Better that than unexpected weird behaviour that we were getting before though because of the undefined params
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 18:22

Hey everyone, so I'm trying to wrap a RESTful service around a nextflow pipeline. All of my tests are working locally, but the nextflow test script is failing when it runs on travis-ci. The nextflow script is running fine, but it's immediately returning an exit status of zero instead of waiting until the process is done. Therefore, the test fails because the output from the nextflow pipeline hasn't been generated yet.

If I'm running nextflow within code, then do I need to execute it in a certain way for it to not run in the background? Have people run into similar issues before?

Here are links to the travis build log, the github repository and the nextflow script.

Paolo Di Tommaso
@pditommaso
Nov 23 2016 18:28
Weird, this reminds me what wrote @tantrev a few days ago
@tantrev
well what happens if I don't, is that docker for some reason thinks NF stops when it really hasn't. Hence, I've tried to work around it by launching it background and just monitoring to see if NF has actually terminated.
eventually thinks NF stops, it doesn't happen immediately
I've been running rather large jobs with thousands of files though, not sure if that's making a difference
could it be that the NF process is killed because it uses too much memory ?
though it should return a non-zero exit status ..
have u any way to access the .nextflow.log file ?
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 18:34
@pditommaso Yeah, I'll print it out to the console during the travis build so I can see it. I'll post the output when it finishes.
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 18:57
.nextflow.log -- Nov-23 18:53:36.218 [main] DEBUG nextflow.cli.Launcher - $> /home/travis/build/mhalagan-nmdp/service-gfe-submission/nextflow run /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/t/../bin/hml.nf --hml /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/t/resources/hmltest1.HML --output /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission --name 922
.nextflow.log -- Nov-23 18:53:36.597 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 0.22.5
.nextflow.log -- Nov-23 18:53:36.641 [main] INFO nextflow.cli.CmdRun - Launching /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/t/../bin/hml.nf [distracted_booth] - revision: 050a11cd56
.nextflow.log -- Nov-23 18:53:37.584 [main] DEBUG nextflow.Session - Session uuid: 6cde7e2a-09b0-4ed3-8b0e-0d46e56310e9
.nextflow.log -- Nov-23 18:53:37.585 [main] DEBUG nextflow.Session - Run name: distracted_booth
.nextflow.log -- Nov-23 18:53:37.588 [main] DEBUG nextflow.Session - Executor pool size: 2
.nextflow.log -- Nov-23 18:53:37.633 [main] DEBUG nextflow.cli.CmdRun -
.nextflow.log -- Version: 0.22.5 build 4102
.nextflow.log -- Modified: 17-11-2016 10:46 UTC
.nextflow.log -- System: Linux 3.13.0-92-generic
.nextflow.log -- Runtime: Groovy 2.4.7 on Java HotSpot(TM) 64-Bit Server VM 1.7.0_80-b15
.nextflow.log -- Encoding: UTF-8 (UTF-8)
.nextflow.log -- Process: 14884@testing-gce-c63d3f36-07ce-4c35-b3ef-432cf195fa58 [127.0.1.1]
.nextflow.log -- CPUs: 2 - Mem: 7.3 GB (5.7 GB) - Swap: 0 (0)
.nextflow.log -- Nov-23 18:53:37.690 [main] DEBUG nextflow.Session - Work-dir: /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/work [ext2/ext3]
.nextflow.log -- Nov-23 18:53:37.691 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/bin/bin
.nextflow.log -- Nov-23 18:53:37.795 [main] DEBUG nextflow.Session - Session start invoked
.nextflow.log -- Nov-23 18:53:37.804 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
.nextflow.log -- Nov-23 18:53:37.804 [main] DEBUG nextflow.script.ScriptRunner - > Script parsing
.nextflow.log -- Nov-23 18:53:38.061 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
.nextflow.log -- Nov-23 18:53:38.301 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: null
.nextflow.log -- Nov-23 18:53:38.301 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
.nextflow.log -- Nov-23 18:53:38.331 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
.nextflow.log -- Nov-23 18:53:38.334 [main] INFO nextflow.executor.Executor - [warm up] executor > local
.nextflow.log -- Nov-23 18:53:38.350 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=2; memory=7.3 GB; capacity=2; pollInterval=100ms; dumpInterval=5m
.nextflow.log -- Nov-23 18:53:38.357 [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: LocalPollingMonitor
.nextflow.log -- Nov-23 18:53:38.358 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
.nextflow.log -- Nov-23 18:53:38.373 [main] DEBUG nextflow.executor.Executor - Invoke register for executor: local
.nextflow.log -- Nov-23 18:53:38.586 [main] DEBUG nextflow.Session - >>> barrier register (process: extractConsensus)
.nextflow.log -- Nov-23 18:53:38.605 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > extractConsensus -- maxForks: 2
.nextflow.log -- Nov-23 18:53:38.757 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <extractConsensus> Poison pill arrived
.nextflow.log -- Nov-23 18:53:38.760 [Actor Thread 2] DEBUG nextflow.processor.StateObj - <extractConsensus> State before poison: StateObj[submitted: 1; completed: 0; poisoned: false ]
.nextflow.log -- Nov-23 18:53:38.786 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: null
.nextflow.log -- Nov-23 18:53:38.
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 19:02
@pditommaso So it looks like it's using 5.7 GB of the 7.3 available memory. From the logs does it seem to be a memory issue?
Paolo Di Tommaso
@pditommaso
Nov 23 2016 19:36
hard to say the log does not terminate correclty
try to define this var in travis env
NXF_OPTS='-Xms512M -Xmx2G'
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 19:42
@pditommaso Sorry, here's the rest of the log:
nextflow.log -- Nov-23 18:53:38.786 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: null
.nextflow.log -- Nov-23 18:53:38.786 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
.nextflow.log -- Nov-23 18:53:38.786 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
.nextflow.log -- Nov-23 18:53:38.787 [main] DEBUG nextflow.Session - >>> barrier register (process: getGFE)
.nextflow.log -- Nov-23 18:53:38.818 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > getGFE -- maxForks: 2
.nextflow.log -- Nov-23 18:53:38.889 [main] DEBUG nextflow.script.ScriptRunner - > Await termination
.nextflow.log -- Nov-23 18:53:38.889 [main] DEBUG nextflow.Session - Session await
.nextflow.log -- Nov-23 18:53:39.150 [Actor Thread 3] INFO nextflow.Session - [cc/a0326d] Submitted process > extractConsensus (1)
.nextflow.log -- Nov-23 18:53:39.154 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <extractConsensus> After stop
.nextflow.log -- Nov-23 18:53:41.388 [Actor Thread 2] DEBUG nextflow.processor.TaskProcessor - <extractConsensus> Sending poison pills and terminating process
.nextflow.log -- Nov-23 18:53:41.394 [Actor Thread 2] DEBUG nextflow.Session - <<< barrier arrive (process: extractConsensus)
.nextflow.log -- Nov-23 18:53:41.461 [Actor Thread 4] INFO nextflow.Session - [b7/2f6359] Submitted process > getGFE (1111111111)
.nextflow.log -- Nov-23 18:53:41.474 [Actor Thread 6] INFO nextflow.Session - [f5/3a7923] Submitted process > getGFE (1111111111)
.nextflow.log -- Nov-23 18:53:42.957 [Actor Thread 4] INFO nextflow.Session - [46/3f08c6] Submitted process > getGFE (1111111111)
.nextflow.log -- Nov-23 18:53:43.109 [Actor Thread 3] INFO nextflow.Session - [11/9bb578] Submitted process > getGFE (222222222)
.nextflow.log -- Nov-23 18:53:44.235 [Actor Thread 4] INFO nextflow.Session - [28/1642b6] Submitted process > getGFE (222222222)
.nextflow.log -- Nov-23 18:53:44.361 [Actor Thread 3] INFO nextflow.Session - [fc/f943fa] Submitted process > getGFE (222222222)
.nextflow.log -- Nov-23 18:53:45.491 [Actor Thread 4] INFO nextflow.Session - [d9/d4476c] Submitted process > getGFE (333333333)
.nextflow.log -- Nov-23 18:53:45.605 [Actor Thread 3] INFO nextflow.Session - [6e/acc168] Submitted process > getGFE (333333333)
.nextflow.log -- Nov-23 18:53:46.764 [Actor Thread 4] INFO nextflow.Session - [51/ecde0e] Submitted process > getGFE (333333333)
.nextflow.log -- Nov-23 18:53:46.879 [Actor Thread 3] INFO nextflow.Session - [77/69320c] Submitted process > getGFE (444444444)
.nextflow.log -- Nov-23 18:53:46.885 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <getGFE> Poison pill arrived
.nextflow.log -- Nov-23 18:53:46.887 [Actor Thread 2] DEBUG nextflow.processor.StateObj - <getGFE> State before poison: StateObj[submitted: 12; completed: 8; poisoned: false ]
.nextflow.log -- Nov-23 18:53:48.038 [Actor Thread 4] INFO nextflow.Session - [ae/331cfb] Submitted process > getGFE (444444444)
.nextflow.log -- Nov-23 18:53:48.113 [Actor Thread 3] INFO nextflow.Session - [6f/cb11a6] Submitted process > getGFE (444444444)
.nextflow.log -- Nov-23 18:53:48.114 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <getGFE> After stop
.nextflow.log -- Nov-23 18:53:49.316 [Actor Thread 2] DEBUG nextflow.processor.TaskProcessor - <getGFE> Sending poison pills and terminating process
.nextflow.log -- Nov-23 18:53:49.319 [Actor Thread 2] DEBUG nextflow.Session - <<< barrier arrive (process: getGFE)
.nextflow.log -- Nov-23 18:53:49.320 [main] DEBUG nextflow.Session - Session await > all process finished
.nextflow.log -- Nov-23 18:53:49.359 [Actor Thread 3] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 12; slices: 1; internal sort time: 0.03 s; external sort time: 0.003 s; total time: 0.033 s
.nextflow.log -- Nov-23 18:53:49.359 [Actor Thread 3] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 12; slices: 1; internal sort time: 0.03 s; external sort time: 0.003 s; total time: 0.033 s
.nextflow.log -- Nov-23 18:53:49.368 [Actor Thread 3] DEBUG nextflow.file.FileCollector - Saved collect-files list to: /tmp/003b526dc890385adca4012e7174dcc0.collect-file
.nextflow.log -- Nov-23 18:53:49.369 [Actor Thread 1] INFO nextflow.Nextflow - Copying 922.txt into: /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission
.nextflow.log -- Nov-23 18:53:49.384 [Actor Thread 3] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-3828137826437206920
.nextflow.log -- Nov-23 18:53:49.415 [Thread-1] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local)
.nextflow.log -- Nov-23 18:53:49.416 [main] DEBUG nextflow.Session - Session await > all barriers passed
.nextflow.log -- Nov-23 18:53:49.433 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
Paolo Di Tommaso
@pditommaso
Nov 23 2016 19:43
ah
ignore the previous option the memory
set the following var instead
NXF_VER=0.22.4
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 20:00
Hmm, that didn't seem to work.
N E X T F L O W ~ version 0.22.4
Launching /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/t/../bin/hml.nf [pedantic_davinci] - revision: 050a11cd56
[warm up] executor > local
[4a/718140] Submitted process > extractConsensus (1)
[c3/9d6aab] Submitted process > getGFE (1111111111)
[0e/536bb8] Submitted process > getGFE (1111111111)
[d9/f3edbe] Submitted process > getGFE (222222222)
[01/2fc96f] Submitted process > getGFE (1111111111)
[f3/a2ab47] Submitted process > getGFE (222222222)
[17/d0bec0] Submitted process > getGFE (222222222)
[b7/4d808d] Submitted process > getGFE (333333333)
[13/a60231] Submitted process > getGFE (333333333)
[36/2bbb87] Submitted process > getGFE (333333333)
[2e/0da300] Submitted process > getGFE (444444444)
[e4/a2d3eb] Submitted process > getGFE (444444444)
[f7/c7c037] Submitted process > getGFE (444444444)
Copying 370.txt into: /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission
.nextflow.log -- Nov-23 19:56:46.024 [main] DEBUG nextflow.cli.Launcher - $> /home/travis/build/mhalagan-nmdp/service-gfe-submission/nextflow run /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/t/../bin/hml.nf --hml /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/t/resources/hmltest1.HML --output /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission --name 370
.nextflow.log -- Nov-23 19:56:46.350 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 0.22.4
.nextflow.log -- Nov-23 19:56:46.369 [main] INFO nextflow.cli.CmdRun - Launching /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/t/../bin/hml.nf [pedantic_davinci] - revision: 050a11cd56
.nextflow.log -- Nov-23 19:56:47.431 [main] DEBUG nextflow.Session - Session uuid: a64717b3-c94c-43c0-ac4c-02953c02b281
.nextflow.log -- Nov-23 19:56:47.431 [main] DEBUG nextflow.Session - Run name: pedantic_davinci
.nextflow.log -- Nov-23 19:56:47.435 [main] DEBUG nextflow.Session - Executor pool size: 2
.nextflow.log -- Nov-23 19:56:47.480 [main] DEBUG nextflow.cli.CmdRun -
.nextflow.log -- Version: 0.22.4 build 4101
.nextflow.log -- Modified: 04-11-2016 20:10 UTC
.nextflow.log -- System: Linux 3.13.0-92-generic
.nextflow.log -- Runtime: Groovy 2.4.7 on Java HotSpot(TM) 64-Bit Server VM 1.7.0_80-b15
.nextflow.log -- Encoding: UTF-8 (UTF-8)
.nextflow.log -- Process: 14955@testing-gce-45646c97-d7c2-4ba4-bfef-ffead5cbc2ca [127.0.1.1]
.nextflow.log -- CPUs: 2 - Mem: 7.3 GB (5.7 GB) - Swap: 0 (0)
.nextflow.log -- Nov-23 19:56:47.509 [main] DEBUG nextflow.Session - Work-dir: /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/work [ext2/ext3]
.nextflow.log -- Nov-23 19:56:47.510 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission/bin/bin
.nextflow.log -- Nov-23 19:56:47.611 [main] DEBUG nextflow.Session - Session start invoked
.nextflow.log -- Nov-23 19:56:47.620 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
.nextflow.log -- Nov-23 19:56:47.621 [main] DEBUG nextflow.script.ScriptRunner - > Script parsing
.nextflow.log -- Nov-23 19:56:47.967 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
.nextflow.log -- Nov-23 19:56:48.206 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: null
.nextflow.log -- Nov-23 19:56:48.207 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
.nextflow.log -- Nov-23 19:56:48.219 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
.nextflow.log -- Nov-23 19:56:48.222 [main] INFO nextflow.executor.Executor - [warm up] executor > local
.nextflow.log -- Nov-23 19:56:48.229 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus: 2; memory: 7.3 GB; pollInterval: 100ms; dumpInterval: 5m
.nextflow.log -- Nov-23 19:56:48.236 [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: LocalPollingMonitor
.nextflow.log -- Nov-23 19:56:48.237 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
.nextflow.log -- Nov-23 19:56:48.240 [main] DEBUG nextflow.executor.Executor - Invoke register for executor: local
.nextflow.log -- Nov-23 19:56:48.504 [main] DEBUG nextflow.Session - >>> barrier register (process: extractConsensus)
.nextflow.log -- Nov-23 19:56:48.541 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > extractConsensus -- maxForks: 2
.nextflow.log -- Nov-23 19:56:48.740 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <extractConsensus> Poison pill arrived
.nextflow.log -- Nov-23 19:56:48.742 [Actor Thread 2] DEBUG nextflow.processor.StateObj - <extractConsensus> State before poison: StateObj[submitted: 1; completed: 0; poisoned: false ]
.nextflow.log -- Nov-23 19:56:48.785 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: null
.nextflow.log -- Nov-23 19:56:48.785 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
.nextflow.log -- Nov-23 19:56:48.785 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
.nextflow.log -- Nov-23 19:56:48.787 [main] DEBUG nextflow.Session - >>> barrier register (process: getGFE)
.nextflow.log -- Nov-23 19:56:48.831 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > getGFE -- maxForks: 2
.nextflow.log -- Nov-23 19:56:48.972 [main] DEBUG nextflow.script.ScriptRunner - > Await termination
.nextflow.log -- Nov-23 19:56:48.973 [main] DEBUG nextflow.Session - Session await
.nextflow.log -- Nov-23 19:56:49.188 [Actor Thread 3] INFO nextflow.Session - [4a/718140] Submitted process > extractConsensus (1)
.nextflow.log -- Nov-23 19:56:51.648 [Actor Thread 2] DEBUG nextflow.processor.TaskProcessor - <extractConsensus> Sending poison pills and terminating process
.nextflow.log -- Nov-23 19:56:51.650 [Actor Thread 2] DEBUG nextflow.Session - <<< barrier arrive (process: extractConsensus)
.nextflow.log -- Nov-23 19:56:51.727 [Actor Thread 6] INFO nextflow.Session - [c3/9d6aab] Submitted process > getGFE (1111111111)
.nextflow.log -- Nov-23 19:56:51.729 [Actor Thread 4] INFO nextflow.Session - [0e/536bb8] Submitted process > getGFE (1111111111)
.nextflow.log -- Nov-23 19:56:53.144 [Actor Thread 4] INFO nextflow.Session - [d9/f3edbe] Submitted process > getGFE (222222222)
.nextflow.log -- Nov-23 19:56:53.291 [Actor Thread 6] INFO nextflow.Session - [01/2fc96f] Submitted process > getGFE (1111111111)
.nextflow.log -- Nov-23 19:56:54.414 [Actor Thread 4] INFO nextflow.Session - [f3/a2ab47] Submitted process > getGFE (222222222)
.nextflow.log -- Nov-23 19:56:54.508 [Actor Thread 6] INFO nextflow.Session - [17/d0bec0] Submitted process > getGFE (222222222)
.nextflow.log -- Nov-23 19:56:55.659 [Actor Thread 4] INFO nextflow.Session - [b7/4d808d] Submitted process > getGFE (333333333)
.nextflow.log -- Nov-23 19:56:55.737 [Actor Thread 6] INFO nextflow.Session - [13/a60231] Submitted process > getGFE (333333333)
.nextflow.log -- Nov-23 19:56:56.920 [Actor Thread 7] INFO nextflow.Session - [36/2bbb87] Submitted process > getGFE (333333333)
.nextflow.log -- Nov-23 19:56:56.970 [Actor Thread 6] INFO nextflow.Session - [2e/0da300] Submitted process > getGFE (444444444)
.nextflow.log -- Nov-23 19:56:56.972 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <getGFE> Poison pill arrived
.nextflow.log -- Nov-23 19:56:56.973 [Actor Thread 2] DEBUG nextflow.processor.StateObj - <getGFE> State before poison: StateObj[submitted: 12; completed: 8; poisoned: false ]
.nextflow.log -- Nov-23 19:56:58.162 [Actor Thread 7] INFO nextflow.Session - [e4/a2d3eb] Submitted process > getGFE (444444444)
.nextflow.log -- Nov-23 19:56:58.262 [Actor Thread 3] INFO nextflow.Session - [f7/c7c037] Submitted process > getGFE (444444444)
.nextflow.log -- Nov-23 19:56:58.264 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <getGFE> After stop
.nextflow.log -- Nov-23 19:56:59.426 [Actor Thread 5] DEBUG nextflow.processor.TaskProcessor - <getGFE> Sending poison pills and terminating process
.nextflow.log -- Nov-23 19:56:59.427 [Actor Thread 5] DEBUG nextflow.Session - <<< barrier arrive (process: getGFE)
.nextflow.log -- Nov-23 19:56:59.431 [main] DEBUG nextflow.Session - Session await > processes completed
.nextflow.log -- Nov-23 19:56:59.459 [Actor Thread 1] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 12; slices: 1; internal sort time: 0.021 s; external sort time: 0.004 s; total time: 0.025 s
.nextflow.log -- Nov-23 19:56:59.468 [Actor Thread 1] DEBUG nextflow.file.FileCollector - Saved collect-files list to: /tmp/86302c952bdaf1446f634ffd8fbc11de.collect-file
.nextflow.log -- Nov-23 19:56:59.469 [Actor Thread 6] INFO nextflow.Nextflow - Copying 370.txt into: /home/travis/build/mhalagan-nmdp/service-gfe-submission/gfe_submission
.nextflow.log -- Nov-23 19:56:59.488 [Actor Thread 1] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-3666660774040507218
.nextflow.log -- Nov-23 19:56:59.524 [Thread-1] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local)
.nextflow.log -- Nov-23 19:56:59.525 [main] DEBUG nextflow.Session - Session await > done
.nextflow.log -- Nov-23 19:56:59.544 [main] DEBUG nextflow.Session - Session destroyed
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:01
please use pastebin.com or a similar thing for logs
therefore same problem?
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 20:03
Sorry, thanks for pointing me to pastebin. Yeah, same problem.
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:05
how do you launch it ?
I'm not find nextflow in the travis file
Sandeep Shantharam
@machbio
Nov 23 2016 20:06
@pditommaso is the Kube Executor part of 0.22.5 version ?
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:06
yes
Sandeep Shantharam
@machbio
Nov 23 2016 20:07
I am sorry, what should the executor name be - process.executor=?
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:07
k8s
;)
Sandeep Shantharam
@machbio
Nov 23 2016 20:08
Ok.. Thank you
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:08
(sorry, not yet documented because it's still experimental)
Sandeep Shantharam
@machbio
Nov 23 2016 20:08
Ya, I understand.. I will test it out this holiday ..
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:08
:+1:
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 20:11

This is in the .travis.yml (line 19)

curl -fsSL get.nextflow.io | bash

Running the test launches the nextflow script. (line 33)

PERL5OPT=-MDevel::Cover=-coverage,statement,branch,condition,path,subroutine prove -lrsv t/022_flow_hml.t

Here's how it's executed within the application:

my @args = ($s_nextflow_cmd, " --hml $s_hml_file"," --output ".$self->outdir," --name ".$self->fileid," &> ".$s_nextflow_log);
my $exit_value      = system(join("",@args));
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:12
ah, it's invoked by a perl script
BTW from the log it seems the pipeline run successfully. What's wrong in the execution ?
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 20:20
Yeah, it runs fine it just returns an exit status right away so everything fails downstream because the output hasn't been generated yet.
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:26
this is the snippet that's supposed to produce the final output, right?
Michael Halagan
@mhalagan-nmdp
Nov 23 2016 20:32
Yeah, the above system command executes nextflow and should produce an output file that is then parsed and returned as a json object.
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:33
ok, in the log I see Copying 922.txt into: /home/travis...
thus it enters in the copy method here
you may want to add another log.info just after file.copyTo(outputDir) to verify if the copy completes
Sandeep Shantharam
@machbio
Nov 23 2016 20:45
how does nextflow know what kubernette cluster to run on - do I need to configure any variable ?
Paolo Di Tommaso
@pditommaso
Nov 23 2016 20:47
NF access the cluster by using kubectl