Does that allow me to reuse
oneFile.first() as input to other processes too?
nearly
you should assign assign to a variable and then use that variable as input eg
oneFile_ch = oneFile.first()
process foo {
input:
file x from oneFile_ch
}
process bar {
input:
file x from oneFile_ch
}
Does this block at
oneFile_ch = ... until it is available?
was always there
Cool!
What's the word on the feature to split up scripts in multiple files? And what about having namespaces?
the current status is described here https://github.com/skptic/nextflow/blob/master/docs/modularity.rst
In the Template examples, it refers to
templates/mapping_kallisto.nf - should this instead be .sh?
it will be executed according the shebang declaration you use (bash by default), extension is not taken in consideration
It's just that the lines before reference
.sh files - I was confused at first why there were two different files with almost the same name.
nope, the first line in the template script
To clarify, I understand how shebangs work. The docs mention both
mapping_kallisto.sh and mapping_kallisto.nf, which I think is a potential source of confusion for readers.
I don't remember that, can you please share the link ?
https://github.com/skptic/nextflow/blob/master/docs/modularity.rst, under Templates
ahhh, that's an unofficial documentation, sorry for that
Ah, I see, thank you.
that's why it's not yet merged in the main docs
you are welcome, thanks for pointing it out
Hi @pditommaso do you know anything about how slurm associates JobIds with tasks? I've run into a situation where I see a process hanging in nextflow:
~> TaskHandler[jobId: 11762; id: 11728; name: splitInfernalFastaBySize (4888); status: SUBMITTED; exit: -; error: -; workDir: /mnt/efs/nextflow/run.0a0a85ee-f4cb-4c90-ae7e-ce277ba9e016/work/18/c05f904d729e317733bf842686acd7 started: -; exited: -; ]Now, if I look in the slurm job_comp.log for that same job id, I find this entry:
JobId=11762 UserId=ubuntu(1000) GroupId=ubuntu(1000) Name=nf-splitInfernalFastaBySize_(4893) JobState=COMPLETED Partition=normal TimeLimit=UNLIMITED StartTime=2017-11-06T08:54:38 EndTime=2017-11-06T08:54:40 NodeList=ip-172-20-22-86 NodeCnt=1 ProcCnt=1 WorkDir=/mnt/efs/nextflow/run.0a0a85ee-f4cb-4c90-ae7e-ce277ba9e016/work/07/1a9b9b6be2811383bcbf3c48db4c21However, if you look closely you'll see that what's in the job_comp.log is for a different task id (4893 instead of 4888) and the work dir is different. In the slurmctl.log it looks like task 4888 is allocated to job id 11762 first, but before the job runs, task 4893 comes along and IT then seems to get assigned to job id 11762. Have you ever seen anything like this? Any idea what might be going on?
weird
the slurm jobId is taken from the sbatch output
then set here
that's the only place the the
jobId attribute is written, so I don't see any potential race condition that mess-up things
Yeah, I'm almost certain this is a race condition on the slurm side and that slurm is returning the same job id for two jobs and in the process losing one!
ugly !
frankly never seen something like that, and it would be surprising ..
Yeah, very strange. Although this pipeline spawns a LOT of jobs. I need to spend some time with Google to see if anyone else has seen anything like this.
keep me posted
Will do. I may just see about upgrading the version of slurm I'm using...