These are chat archives for nextflow-io/nextflow

1st
Jul 2016
Jacek Grzebyta
@jgrzebyta
Jul 01 2016 09:10
@pditommaso Hi, I have produced example to my request about joining workflows. I added as pull to the examples project: nextflow-io/examples#2
Paolo Di Tommaso
@pditommaso
Jul 01 2016 14:23
@jgrzebyta Thanks, I will give it a try at my first convenience
@mes5k ok, let me know how I can help you
Mike Smoot
@mes5k
Jul 01 2016 15:56
@pditommaso Hi Paolo, so I've got a pipeline reliably producing the underflow exception and I know how to make the exception go away, but I don't understand the root cause of the exception. What debugging can I do to help figure this out?
Paolo Di Tommaso
@pditommaso
Jul 01 2016 15:57
Hi Mike, let me check the stack trace again
um, it's related to serialisation/deserialisation process
you could try to debug around the line nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy:510)
do you have a test case that replicate the error?
Mike Smoot
@mes5k
Jul 01 2016 16:02
No test case, just a pipeline that fails reliably.
Mike Smoot
@mes5k
Jul 01 2016 16:18
Well, adding -trace nextflow.processor.TaskProcessor has slowed things down substantially. My suspicion given that the exception is about underflow is that the loop it not ever breaking. If that's the case then I should see a whole bunch of log messages.
Paolo Di Tommaso
@pditommaso
Jul 01 2016 16:21
and how have you managed to fix it?
Mike Smoot
@mes5k
Jul 01 2016 16:27
Fixing was a matter of renaming things - I'll see if I can show you a diff in a minute. In the meantime, bit looks like the .command.val file is empty, so the file that TaskContext is trying to read at TaskProcessor line 622. Is that right?
Paolo Di Tommaso
@pditommaso
Jul 01 2016 16:29
yes, if .command.val is empty that could be the reason
could = is
Mike Smoot
@mes5k
Jul 01 2016 16:30
Ok, great. So it would be easy to check for an empty file, but is having an empty file OK?
Paolo Di Tommaso
@pditommaso
Jul 01 2016 16:32
actually that file should not be written when there isn't a context to serialise
Here is where that file is saved
Mike Smoot
@mes5k
Jul 01 2016 16:35
Another bit of information: for the task that fails, I do see the [${task.name}] Cacheable folder: $folder -- exists: $exists; try: $tries log message twice
Paolo Di Tommaso
@pditommaso
Jul 01 2016 16:37
the exception is raised during the deserialisation, but my guess is that the bug is in the serialisation step, when the file is written
Mike Smoot
@mes5k
Jul 01 2016 16:37
Yup, that makes sense.
Is it worth putting checks into the deserialization phase?
Paolo Di Tommaso
@pditommaso
Jul 01 2016 16:41
I would put a warning
because that should not happen, and something in the resume won't work as expected
(need to go)
Mike Smoot
@mes5k
Jul 01 2016 16:42
Sure. Let me know if you'd like me to submit a patch for the checks or whether it's easier for you to do.
I'm not sure how to handle the serialization phase of things.
Mike Smoot
@mes5k
Jul 01 2016 16:50
This fix works for me
diff --git a/src/main/groovy/nextflow/processor/TaskProcessor.groovy b/src/main/groovy/nextflow/proc
index 427c9d1..98072b1 100644
--- a/src/main/groovy/nextflow/processor/TaskProcessor.groovy
+++ b/src/main/groovy/nextflow/processor/TaskProcessor.groovy
@@ -621,8 +621,8 @@ abstract class TaskProcessor {
             try {
                 ctx = TaskContext.read(this, ctxFile)
             }
-            catch( IOException e ) {
-                log.trace "[$task.name] Context map can't be read: $ctxFile -- return false -- Caus
+            catch( Exception e ) {
+                log.debug "[$task.name] Context map can't be read: $ctxFile -- return false -- Caus
                 return false
             }
         }
Jacek Grzebyta
@jgrzebyta
Jul 01 2016 23:27
@pditommaso So finally I added more stuff to my example: communication between the workflows using a json file.