These are chat archives for nextflow-io/nextflow

21st
Jun 2018
spaceturtle
@spaceturtle
Jun 21 2018 02:44
What happened when the process reached the maximum number of retry? Would the whole pipeline terminate or just ignore this process and continue all other jobs?
Simone Baffelli
@baffelli
Jun 21 2018 06:38

Is it normal that the nextflow process just keeps on living if all tasks are completed? The last lines of the log say

Jun-20 19:28:27.302 [main] DEBUG nextflow.Session - Session await > all process finished
Jun-20 19:28:27.302 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local)
Jun-20 19:28:27.303 [main] DEBUG nextflow.Session - Session await > all barriers passed

I presume no operator or process is waiting for any values, so why would the main nextflow process not exit? Is it by design?

Paolo Di Tommaso
@pditommaso
Jun 21 2018 06:41
There should be an active operator, if you kill it with ctrl+c it dumps the active ones in the log file
@spaceturtle it stops
Simone Baffelli
@baffelli
Jun 21 2018 06:42
Yes indeed, many operators were active; thanks.
But this begs the question: how to determine why an operator is active?
Paolo Di Tommaso
@pditommaso
Jun 21 2018 06:43
Messing up some channel create/close ?
Simone Baffelli
@baffelli
Jun 21 2018 06:44
What do you mean?
Paolo Di Tommaso
@pditommaso
Jun 21 2018 06:45
do you use any Channel.create ?
Simone Baffelli
@baffelli
Jun 21 2018 06:46
Two!
I wasn't aware that it was a problem
Paolo Di Tommaso
@pditommaso
Jun 21 2018 06:47
it's a problem if it's misused ie. not closed properly
and in general and don't suggest to use it
Simone Baffelli
@baffelli
Jun 21 2018 06:48
I think I could change some parts and not use Channel.create
thanks for the suggestion
Paolo Di Tommaso
@pditommaso
Jun 21 2018 06:50
:+1:
Simone Baffelli
@baffelli
Jun 21 2018 06:53
Is there some undocumented version of choice that allows me avoiding Channel.create?
Simone Baffelli
@baffelli
Jun 21 2018 07:12
Nvm, solved that. But many operators are still active
Paolo Di Tommaso
@pditommaso
Jun 21 2018 08:17
Do u mean it hangs?
Simone Baffelli
@baffelli
Jun 21 2018 08:41
Yeah
Paolo Di Tommaso
@pditommaso
Jun 21 2018 09:04
this was happening in the past, right ?
identify the changes compared to the last working version ..
Simone Baffelli
@baffelli
Jun 21 2018 10:09
It's been happening for a long while
Paolo Di Tommaso
@pditommaso
Jun 21 2018 10:13
check the first process in active status in the log status when you stop the execution
the problem in one of the input channel or a upstream operator
Simone Baffelli
@baffelli
Jun 21 2018 10:17
That's a good tip
In fact i was trying to understand where to look upstram
micans
@micans
Jun 21 2018 12:25
Is it possible to take action (in my case logging) in case of an error (in my case one that is ignored). The idea is to create a summary of which samples failed at which stage. So something like an onErorr: directive. Alternatively I could parse the nextflow log afterwards. report.html is close, but it would be nice to be able to do customisation and push/publish custom files. We are running the pipeline on thousands of samples, and with this quantity of data we find it useful to (sometimes) ignore errors at some stages as new types of errors keep popping up. Thoughts most welcome.
Paolo Di Tommaso
@pditommaso
Jun 21 2018 12:27
this could be an interesting improvement, please open a feature request elaborating your use case
micans
@micans
Jun 21 2018 12:28
Cheers, will do
Simone Baffelli
@baffelli
Jun 21 2018 13:29
@pditommaso there are no active processes when I stop the execution, only operators
Simone Baffelli
@baffelli
Jun 21 2018 13:47
The last message I see before I manually interrput is: Jun-21 15:46:02.511 [main] DEBUG nextflow.Session - Session await > all barriers passed
Paolo Di Tommaso
@pditommaso
Jun 21 2018 13:47
bisect your your script until you isolate the problem, it could operator misuse or even a bug
Simone Baffelli
@baffelli
Jun 21 2018 13:49
:worried: my script is enormous
It would be useful to have an ID attached to the operators
so I could find out which of the operators is waiting
Simone Baffelli
@baffelli
Jun 21 2018 14:51
Can an unused channel cause it_
?
Paolo Di Tommaso
@pditommaso
Jun 21 2018 14:57
no
micans
@micans
Jun 21 2018 14:58
I've thought more about error actions - and concluded that my idea is too hazy, it sat inbetween injecting custom logging into nextflow and hoping to utilise nextflow channels for this (an abominable idea, but I found it at the back of my mind). I'll start with trace.txt parsing, and have noted that the task section in report.html is great. It may be nice to have as a tsv text file to create summary reports from.
Shellfishgene
@Shellfishgene
Jun 21 2018 15:28

I want to create a channel from file pairs, duplicate it, and then flatten one of those to have all files in one command.

    .fromFilePairs( params.reads )
    .ifEmpty { error "Cannot find any reads matching: ${params.reads}" }
    .into { read_files_fastqc; read_files_fastqc2 } 

flattened = read_files_fastqc2.flatMap { key, files -> [[key], files].combinations() }.collectFile()

This does not work, not sure why...

WARN: Input tuple does not match input set cardinality declared by process is the error
Solve the warning, that was in the process of course...
Caspar
@caspargross
Jun 21 2018 18:30
Nextflow crashes with a StackOverFlow error. When I manually execute the process in the work folder everything runs fine. Is this a problem with nextflow or with my tools/script ? Here is a part of the log file
[fc/b41030] Submitted process > delly (S1221)
Jun-21 19:59:07.155 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jun-21 19:59:07.155 [Task submitter] INFO  nextflow.Session - [67/bdd760] Submitted process > delly (S1404)
Jun-21 19:59:09.084 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 46; name: delly (S1221); status: COMPLETED; exit: 0; error: -; workDir: /mnt/SRV018/users/ahgrosc1/analysis/shortread_plC/work/fc/b4103020729bf94d9dceb28c7999c4]
Jun-21 19:59:09.088 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jun-21 19:59:09.088 [Task submitter] INFO  nextflow.Session - [cf/411aeb] Submitted process > delly (BK3937)
Jun-21 19:59:09.127 [Task monitor] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
java.lang.StackOverflowError: null
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1765)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3758)
    at nextflow.script.WorkflowMetadata.getProperty(WorkflowMetadata.groovy)
    at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:174)
    at nextflow.script.WorkflowMetadata.get(WorkflowMetadata.groovy:284)
    at sun.reflect.GeneratedMethodAccessor1224.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1850)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3758)
    at nextflow.script.WorkflowMetadata.getProperty(WorkflowMetadata.groovy)
    at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:174)
    at nextflow.script.WorkflowMetadata.get(WorkflowMetadata.groovy:284)
    at sun.reflect.GeneratedMethodAccessor1224.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1850)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3758)
    at nextflow.script.WorkflowMetadata.getProperty(WorkflowMetadata.groovy)
    at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:174)
    at nextflow.script.WorkflowMetadata.get(WorkflowMetadata.groovy:284)
    at sun.reflect.GeneratedMethodAccessor1224.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1850)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3758)
    at nextflow.script.WorkflowMetadata.getProperty(WorkflowMetadata.groovy)
    at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:174)
    at nextflow.script.WorkflowMetadata.get(WorkflowMetadata.groovy:284)
    at sun.reflect.GeneratedMethodAccessor1224.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)