These are chat archives for nextflow-io/nextflow

7th
Nov 2017
Tim Diels
@timdiels
Nov 07 2017 11:09 UTC
Does that allow me to reuse oneFile.first() as input to other processes too?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 11:11 UTC
nearly
you should assign assign to a variable and then use that variable as input eg
oneFile_ch = oneFile.first() 

process foo {
  input:
  file x from oneFile_ch
}

process bar {
  input:
  file x from oneFile_ch
}
Tim Diels
@timdiels
Nov 07 2017 11:15 UTC
Does this block at oneFile_ch = ... until it is available?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 11:16 UTC
was always there
Tim Diels
@timdiels
Nov 07 2017 11:49 UTC
Cool!
Tim Diels
@timdiels
Nov 07 2017 11:58 UTC
What's the word on the feature to split up scripts in multiple files? And what about having namespaces?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 12:58 UTC
Jemma Nelson
@fwip
Nov 07 2017 18:11 UTC
In the Template examples, it refers to templates/mapping_kallisto.nf - should this instead be .sh?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 18:13 UTC
it will be executed according the shebang declaration you use (bash by default), extension is not taken in consideration
Jemma Nelson
@fwip
Nov 07 2017 19:32 UTC
It's just that the lines before reference .sh files - I was confused at first why there were two different files with almost the same name.
Paolo Di Tommaso
@pditommaso
Nov 07 2017 19:34 UTC
nope, the first line in the template script
Jemma Nelson
@fwip
Nov 07 2017 19:37 UTC
To clarify, I understand how shebangs work. The docs mention both mapping_kallisto.sh and mapping_kallisto.nf, which I think is a potential source of confusion for readers.
Paolo Di Tommaso
@pditommaso
Nov 07 2017 19:38 UTC
I don't remember that, can you please share the link ?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 19:39 UTC
ahhh, that's an unofficial documentation, sorry for that
Jemma Nelson
@fwip
Nov 07 2017 19:39 UTC
Ah, I see, thank you.
Paolo Di Tommaso
@pditommaso
Nov 07 2017 19:40 UTC
that's why it's not yet merged in the main docs
you are welcome, thanks for pointing it out
Mike Smoot
@mes5k
Nov 07 2017 19:56 UTC

Hi @pditommaso do you know anything about how slurm associates JobIds with tasks? I've run into a situation where I see a process hanging in nextflow:

~> TaskHandler[jobId: 11762; id: 11728; name: splitInfernalFastaBySize (4888); status: SUBMITTED; exit: -; error: -; workDir: /mnt/efs/nextflow/run.0a0a85ee-f4cb-4c90-ae7e-ce277ba9e016/work/18/c05f904d729e317733bf842686acd7 started: -; exited: -; ]

Now, if I look in the slurm job_comp.log for that same job id, I find this entry:

JobId=11762 UserId=ubuntu(1000) GroupId=ubuntu(1000) Name=nf-splitInfernalFastaBySize_(4893) JobState=COMPLETED Partition=normal TimeLimit=UNLIMITED StartTime=2017-11-06T08:54:38 EndTime=2017-11-06T08:54:40 NodeList=ip-172-20-22-86 NodeCnt=1 ProcCnt=1 WorkDir=/mnt/efs/nextflow/run.0a0a85ee-f4cb-4c90-ae7e-ce277ba9e016/work/07/1a9b9b6be2811383bcbf3c48db4c21

However, if you look closely you'll see that what's in the job_comp.log is for a different task id (4893 instead of 4888) and the work dir is different. In the slurmctl.log it looks like task 4888 is allocated to job id 11762 first, but before the job runs, task 4893 comes along and IT then seems to get assigned to job id 11762. Have you ever seen anything like this? Any idea what might be going on?

Paolo Di Tommaso
@pditommaso
Nov 07 2017 20:02 UTC
weird
the slurm jobId is taken from the sbatch output
then set here
that's the only place the the jobId attribute is written, so I don't see any potential race condition that mess-up things
Mike Smoot
@mes5k
Nov 07 2017 20:07 UTC
Yeah, I'm almost certain this is a race condition on the slurm side and that slurm is returning the same job id for two jobs and in the process losing one!
Paolo Di Tommaso
@pditommaso
Nov 07 2017 20:08 UTC
ugly !
frankly never seen something like that, and it would be surprising ..
Mike Smoot
@mes5k
Nov 07 2017 20:10 UTC
Yeah, very strange. Although this pipeline spawns a LOT of jobs. I need to spend some time with Google to see if anyone else has seen anything like this.
Paolo Di Tommaso
@pditommaso
Nov 07 2017 20:10 UTC
keep me posted
Mike Smoot
@mes5k
Nov 07 2017 20:11 UTC
Will do. I may just see about upgrading the version of slurm I'm using...