These are chat archives for nextflow-io/nextflow

18th
May 2018
tbugfinder
@tbugfinder
May 18 2018 06:17
@mes5k @pditommaso For testing purposes I set queueSize to 5 so only 5 jobs were active (had a typo previously). After that I changed it to 50000. Now hundreds of jobs are submitted successfully however job-submission is too slow (23min for ~1500 jobs - one job per second) for my use-case. Any idea for tweaking job-submission?
Just seen this error:
] ERROR nextflow.processor.TaskProcessor - Error executing process > 'executeme'

Caused by:
  Too Many Requests (Service: AWSBatch; Status Code: 429; Error Code: TooManyRequestsException; Request ID: idididididididid)

com.amazonaws.services.batch.model.AWSBatchException: Too Many Requests (Service: AWSBatch; Status Code: 429; Error Code: TooManyRequestsException; Request ID: idididididididid
)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1587)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
        at com.amazonaws.services.batch.AWSBatchClient.doInvoke(AWSBatchClient.java:1252)
        at com.amazonaws.services.batch.AWSBatchClient.invoke(AWSBatchClient.java:1228)
        at com.amazonaws.services.batch.AWSBatchClient.executeSubmitJob(AWSBatchClient.java:1026)
Paolo Di Tommaso
@pditommaso
May 18 2018 06:21
ohh, interesting
a new submission rate limit feature is under testing
I think this could help to prevent your problem.
what do you think ?
BTW could you include the full error stack trace, I don't see the NF related part
tbugfinder
@tbugfinder
May 18 2018 06:26
I'm trying to figure out if this limit can be increased on AWS public API side.
] ERROR nextflow.processor.TaskProcessor - Error executing process > 'executeme'

Caused by:
  Too Many Requests (Service: AWSBatch; Status Code: 429; Error Code: TooManyRequestsException; Request ID: idididididididid)

com.amazonaws.services.batch.model.AWSBatchException: Too Many Requests (Service: AWSBatch; Status Code: 429; Error Code: TooManyRequestsException; Request ID: idididididididid
)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1587)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
        at com.amazonaws.services.batch.AWSBatchClient.doInvoke(AWSBatchClient.java:1252)
        at com.amazonaws.services.batch.AWSBatchClient.invoke(AWSBatchClient.java:1228)
        at com.amazonaws.services.batch.AWSBatchClient.executeSubmitJob(AWSBatchClient.java:1026)
        at com.amazonaws.services.batch.AWSBatchClient.submitJob(AWSBatchClient.java:1002)
        at nextflow.executor.AwsBatchTaskHandler.submit(AwsBatchExecutor.groovy:360)
        at nextflow.processor.TaskPollingMonitor.submit(TaskPollingMonitor.groovy:201)
        at nextflow.processor.TaskPollingMonitor.submitPendingTasks(TaskPollingMonitor.groovy:481)
        at nextflow.processor.TaskPollingMonitor.submitLoop(TaskPollingMonitor.groovy:311)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1092)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
        at groovy.lang.Closure.call(Closure.java:418)
        at groovy.lang.Closure.call(Closure.java:412)
        at groovy.lang.Closure.run(Closure.java:499)
        at java.lang.Thread.run(Thread.java:748)
Paolo Di Tommaso
@pditommaso
May 18 2018 06:27
But I think throttling the submission rate should solve the problem
if you wan to try it use the version 0.30.0-BETA1
tbugfinder
@tbugfinder
May 18 2018 06:30
Well, actually I would like to submit as many jobs as possible immediately. As per my understanding AWS limits the sum of all API requests - for nextflow this would be the submission and the status check. So I'd have to choose a longer interval for job status check and limit submission to 300jobs/second or so. Will give it a try.
Paolo Di Tommaso
@pditommaso
May 18 2018 06:32
status check is aggregated, in the meaning it does not submit a separate request for each job
300jobs/second
how is expected average duration of job in your pipeline ?
tbugfinder
@tbugfinder
May 18 2018 06:37
It is up to us to shrink item input data so that a single job takes 30min - 60min. By scaling horizontally the overall computation should finish within 60min.
Can nextflow update itself to a beta version ?
Paolo Di Tommaso
@pditommaso
May 18 2018 06:38
yep, use NXF_VER=0.30.0-BETA1 nextflow run .. etc
another possibility could be to manage that exception on NF side an automatically throttle the submit requests
Paolo Di Tommaso
@pditommaso
May 18 2018 06:44
@fstrozzi any experience with the above issue ?
Paolo Di Tommaso
@pditommaso
May 18 2018 06:53
@tbugfinder asked to the Batch product manager :)
oh, I've just seen that he is not working anymore at Amazon !
Francesco Strozzi
@fstrozzi
May 18 2018 08:28
@pditommaso @tbugfinder we have not experienced this issue recently, although we received a similar message in the past but that was before NF was updated to group together API calls to AWS Batch. Our largest workload so far was above 4000 jobs submitted, but we plan to have larger workloads soon so I will be able to test this again
Paolo Di Tommaso
@pditommaso
May 18 2018 08:28
:+1:
Francesco Strozzi
@fstrozzi
May 18 2018 08:33
I am not aware of limits on the api calls for batch, at least they are not listed in the documentation but according to api calls / second limits for other services, I think staying under 1000 api calls per second should work fine
Paolo Di Tommaso
@pditommaso
May 18 2018 08:35
I guess some limit to protect agains burst or DDoS there should be
tbugfinder
@tbugfinder
May 18 2018 10:07
@fstrozzi How long did it take to submit your 4000 jobs?
Luca Cozzuto
@lucacozzuto
May 18 2018 10:08
Hi @pditommaso when I define an output like this:
    output:
    set "${cond_A}_${cond_B}", file("${cond_A}.psi"), file("${cond_B}.psi"), file("${cond_A}.tpm"), file("${cond_B}.tpm") into single_output_files
Francesco Strozzi
@fstrozzi
May 18 2018 10:09
@tbugfinder Few minutes if I remember correctly, I was using nextflow 0.27 and it was a couple of months ago
Luca Cozzuto
@lucacozzuto
May 18 2018 10:09
why is Nextflow complaining that it does not find the file "${condA}${cond_B}"?
tbugfinder
@tbugfinder
May 18 2018 10:09
@pditommaso I restarted the job submission using the beta version and 400jobs/s limit - so far it didn't error out. However job-submission still is slow. Could it be an nextflow limit?
May-18 09:12:36.039 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor awsbatch > tasks to be completed: 439 -- pending tasks are shown below
May-18 09:18:28.043 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor awsbatch > tasks to be completed: 805 -- pending tasks are shown below
May-18 09:24:20.087 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor awsbatch > tasks to be completed: 1171 -- pending tasks are shown below
May-18 09:30:12.089 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor awsbatch > tasks to be completed: 1524 -- pending tasks are shown below
May-18 09:36:26.515 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor awsbatch > tasks to be completed: 1882 -- pending tasks are shown below
Luca Cozzuto
@lucacozzuto
May 18 2018 10:13
solved using
    output:
    set val("${cond_A}_${cond_B}"), file("${cond_A}.psi"), file("${cond_B}.psi"), file("${cond_A}.tpm"), file("${cond_B}.tpm") into single_output_files
Paolo Di Tommaso
@pditommaso
May 18 2018 11:38
@tbugfinder do you mean the submission rate is lower than 400/sec
tbugfinder
@tbugfinder
May 18 2018 11:41
@pditommaso exactly
Paolo Di Tommaso
@pditommaso
May 18 2018 11:43
it can also depends by graph of task dependencies in your pipeline
or do you have task with no deps that is supposed to be fired immediately ?
tbugfinder
@tbugfinder
May 18 2018 11:49
There is no pre-dependency and therefore the jobs should start immediately.
Paolo Di Tommaso
@pditommaso
May 18 2018 11:49
I see, what you can do is to enable trace level logging to have a better idea where time is spent.
use the following command
NXF_VER=0.30.0-BETA1 nextflow -trace nextflow.executor run .. etc
than share the resulting file, eventually in a github issue
Bioninbo
@Bioninbo
May 18 2018 12:13

Hello,
How to create custom variable per process in the config file without warnings? It works when I use the following command:

process$process1.custom_variable = 'var'

but I get the following warning message:

WARN: Unknown directive `custom_variable` for process `process1`
Paolo Di Tommaso
@pditommaso
May 18 2018 12:15
use process sectors with label annotation
(requires version 0.29.x)
Bioninbo
@Bioninbo
May 18 2018 12:36
I still get the warning message using withLabel or withName. It is not a big issue though
Paolo Di Tommaso
@pditommaso
May 18 2018 12:37
withLabel should not display any warning
Bioninbo
@Bioninbo
May 18 2018 12:39
it does though (when I comment it I don't get the warning anymore)
I use:
withLabel: label1 {
  current_variable = "var"
}
Paolo Di Tommaso
@pditommaso
May 18 2018 12:40
what is supposed to be current_variable ?
Bioninbo
@Bioninbo
May 18 2018 12:41
a path to use in publishDir
eg:
withLabel: label1 {
  current_path = "metrics/library_complexity"
}
Paolo Di Tommaso
@pditommaso
May 18 2018 12:42
provide a real example with exact warning message
Bioninbo
@Bioninbo
May 18 2018 12:45

Ok. In the config file:

process {
  withLabel: label1 {
    current_path = "metrics/library_complexity"
    container = params.picard
  }
}

in the nf file:

process library_complexity {
  label 'label1'
  publishDir path: "${outdir}/${current_path}", mode: "${publishMode}"
  ...

error message:

WARN: Unknown directive `current_path` for process `library_complexity`
tbugfinder
@tbugfinder
May 18 2018 12:46
@pditommaso do you mean -trace or -with-trace ? Should a file nextflow.executor be created?
Bioninbo
@Bioninbo
May 18 2018 12:46
but the output files are in the correct folder
Paolo Di Tommaso
@pditommaso
May 18 2018 12:46
@tbugfinder I mean -trace using exactly the command line provided
@Bioninbo that syntax is not valid
custom var cannot be defined in the configuration process scope
Bioninbo
@Bioninbo
May 18 2018 12:49
I see. This script is working though somehow despite the warning message
Paolo Di Tommaso
@pditommaso
May 18 2018 12:50
current_path will be null in your script
tbugfinder
@tbugfinder
May 18 2018 12:58
@pditommaso I don't see specific trace output.
Paolo Di Tommaso
@pditommaso
May 18 2018 12:59
it produces a more detailed .nextflow.log file
Bioninbo
@Bioninbo
May 18 2018 13:10
I don't get it, how can current_path be null if the files are published in the correct folder? The variable should be stored somehow
Paolo Di Tommaso
@pditommaso
May 18 2018 13:11
that's a syntax not supported that won't work in future release, my advice is not to use it
Bioninbo
@Bioninbo
May 18 2018 13:11
I see. But then it would be nice to be able to do that. I'd rather set all my paths in the config file. And other variables could be useful too.
Paolo Di Tommaso
@pditommaso
May 18 2018 13:12
use a params.something
Bioninbo
@Bioninbo
May 18 2018 13:13
Yes I can do that for now. But I would prefer to have the same variable to have different values in different process. So I can use the same command in each process and get different outputs
Bioninbo
@Bioninbo
May 18 2018 13:20
But the new process selector feature is really nice thanks for that
Bioninbo
@Bioninbo
May 18 2018 13:42
Is there a way to define a variable in a process? I tried this way but it do not works well:
publishDir path: "${outdir}/${current_path}", mode: "${publishMode}"
input:
 val current_path from "metrics/library_complexity"
Paolo Di Tommaso
@pditommaso
May 18 2018 13:48
that's supposed to work
Bioninbo
@Bioninbo
May 18 2018 13:51
Ah right it works now. I messed up somehow