publishDir path: {params.saveReference ? 'results' : null}, mode: 'copy'
set pair_id, file("${pair_id}*_concat.fq.gz") into {reads; pilon_reads}
set pair_id, "${pair_id}*_concat.fq.gz" into reads, pilon_reads
set val(pair_id), file("${pair_id}*_concat.fq.gz") into (reads, pilon_reads)
Hi @pditommaso I'm finally back to debugging my AWS batch problem, which is that no jobs for my pipeline get submitted for code built from master
whereas the same pipeline submits fine with version 0.30.2
. I'm to the point of diffing nextflow logs. The only difference (apart from not submitting any jobs) is that master
has a throttling executor and a different polling monitor:
main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=50.0qps], poolSize:5, maxPoolSize:5, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
[main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-reaper, limiter:RateLimiter[stableRate=50.0qps], poolSize:5, maxPoolSize:5, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
[main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - Creating parallel monitor for executor 'awsbatch' > pollInterval=10s; dumpInterval=5m
[main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: ParallelPollingMonitor
I've tried reproducing this problem with a simple pipeline to create a test case and I've had no luck. So master
at least works for some pipelines. I'm just not sure where to look next to try and determine what's blocking my pipeline. Any ideas?
My code gets to this point
[main] DEBUG nextflow.executor.Executor - Initializing executor: awsbatch
[main] DEBUG nextflow.Session - >>> barrier register (process: test_mount)
[main] DEBUG nextflow.processor.TaskProcessor - Creating operator > test_mount -- maxForks: 2
[main] DEBUG nextflow.script.ScriptRunner - > Await termination
[main] DEBUG nextflow.Session - Session await
and hangs. In 0.30.2
it immediately follows the Session await
with log messages from the [Task submitter]
NXF_VER=0.30.2
Ok, I've risked life and limb to put this test code up:
${NF} run https://github.com/mes5k/nf-test.git -r master -latest -w s3://super-secret-bucket/nf-test-work
This consistently fails for me with master
and works with 0.30.2
.
nextflow run https://github.com/mes5k/nf-test.git -r master -latest
N E X T F L O W ~ version 0.31.0
Pulling mes5k/nf-test ...
downloaded from https://github.com/mes5k/nf-test.git
Launching `mes5k/nf-test` [loving_colden] - revision: 8a738f4b5a [master]
1
2
3
4
5
[warm up] executor > awsbatch
[ff/985307] Submitted process > create_files (3)
[3a/296bc1] Submitted process > create_files (2)
[3d/0b5f53] Submitted process > create_files (1)
[12/8c45a9] Submitted process > create_files (4)
[fa/801711] Submitted process > create_files (5)
if( cpus == 1 ) then println "error"