Question regarding how jobs are handled when we submit them into our cluster using Slurm: NF does not spawn all jobs at once like other tools with the rationale that is a pipeline execution is not progressing, downstream tasks sit and just way for the previous tasks completion. But this also gives us issues with potentially quite large delays in pipelines.
An easy example: We have a pipeline that firsts submits a job that takes a while to finish (e.g. different variant callers) and at the end there is a quick concatenation of the output of the previous jobs. This last job only take a second or so to run. But while the first jobs were running other users submitted big jobs to the cluster and since NF did not spawn the short concatenation job from the beginning it will be placed last in the queue, delaying it until the queue has been cleared.
One can partially alleviate the problem by using different priorities and such, but in general the behaviour to not spawn jobs in the beginning would disrupt the FIFO idea of a Slurm queue since other tools submit all jobs at once and waits for the job manager to to do its job and prioritize the workload. For me the logical thing would be that you could choose to have the jobs submitted to Slurm from the beginning and that the potentially orphaned jobs that remains in the queue is handled in another manner, but maybe I am missing something of the larger picture as to why NF handles the jobs this way?