These are chat archives for nextflow-io/nextflow

7th
Nov 2017
Tim Diels
@timdiels
Nov 07 2017 11:09
Does that allow me to reuse oneFile.first() as input to other processes too?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 11:11
nearly
you should assign assign to a variable and then use that variable as input eg
oneFile_ch = oneFile.first() 

process foo {
  input:
  file x from oneFile_ch
}

process bar {
  input:
  file x from oneFile_ch
}
Tim Diels
@timdiels
Nov 07 2017 11:15
Does this block at oneFile_ch = ... until it is available?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 11:16
was always there
Tim Diels
@timdiels
Nov 07 2017 11:49
Cool!
Tim Diels
@timdiels
Nov 07 2017 11:58
What's the word on the feature to split up scripts in multiple files? And what about having namespaces?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 12:58
Jemma Nelson
@fwip
Nov 07 2017 18:11
In the Template examples, it refers to templates/mapping_kallisto.nf - should this instead be .sh?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 18:13
it will be executed according the shebang declaration you use (bash by default), extension is not taken in consideration
Jemma Nelson
@fwip
Nov 07 2017 19:32
It's just that the lines before reference .sh files - I was confused at first why there were two different files with almost the same name.
Paolo Di Tommaso
@pditommaso
Nov 07 2017 19:34
nope, the first line in the template script
Jemma Nelson
@fwip
Nov 07 2017 19:37
To clarify, I understand how shebangs work. The docs mention both mapping_kallisto.sh and mapping_kallisto.nf, which I think is a potential source of confusion for readers.
Paolo Di Tommaso
@pditommaso
Nov 07 2017 19:38
I don't remember that, can you please share the link ?
Paolo Di Tommaso
@pditommaso
Nov 07 2017 19:39
ahhh, that's an unofficial documentation, sorry for that
Jemma Nelson
@fwip
Nov 07 2017 19:39
Ah, I see, thank you.
Paolo Di Tommaso
@pditommaso
Nov 07 2017 19:40
that's why it's not yet merged in the main docs
you are welcome, thanks for pointing it out
Mike Smoot
@mes5k
Nov 07 2017 19:56

Hi @pditommaso do you know anything about how slurm associates JobIds with tasks? I've run into a situation where I see a process hanging in nextflow:

~> TaskHandler[jobId: 11762; id: 11728; name: splitInfernalFastaBySize (4888); status: SUBMITTED; exit: -; error: -; workDir: /mnt/efs/nextflow/run.0a0a85ee-f4cb-4c90-ae7e-ce277ba9e016/work/18/c05f904d729e317733bf842686acd7 started: -; exited: -; ]

Now, if I look in the slurm job_comp.log for that same job id, I find this entry:

JobId=11762 UserId=ubuntu(1000) GroupId=ubuntu(1000) Name=nf-splitInfernalFastaBySize_(4893) JobState=COMPLETED Partition=normal TimeLimit=UNLIMITED StartTime=2017-11-06T08:54:38 EndTime=2017-11-06T08:54:40 NodeList=ip-172-20-22-86 NodeCnt=1 ProcCnt=1 WorkDir=/mnt/efs/nextflow/run.0a0a85ee-f4cb-4c90-ae7e-ce277ba9e016/work/07/1a9b9b6be2811383bcbf3c48db4c21

However, if you look closely you'll see that what's in the job_comp.log is for a different task id (4893 instead of 4888) and the work dir is different. In the slurmctl.log it looks like task 4888 is allocated to job id 11762 first, but before the job runs, task 4893 comes along and IT then seems to get assigned to job id 11762. Have you ever seen anything like this? Any idea what might be going on?

Paolo Di Tommaso
@pditommaso
Nov 07 2017 20:02
weird
the slurm jobId is taken from the sbatch output
then set here
that's the only place the the jobId attribute is written, so I don't see any potential race condition that mess-up things
Mike Smoot
@mes5k
Nov 07 2017 20:07
Yeah, I'm almost certain this is a race condition on the slurm side and that slurm is returning the same job id for two jobs and in the process losing one!
Paolo Di Tommaso
@pditommaso
Nov 07 2017 20:08
ugly !
frankly never seen something like that, and it would be surprising ..
Mike Smoot
@mes5k
Nov 07 2017 20:10
Yeah, very strange. Although this pipeline spawns a LOT of jobs. I need to spend some time with Google to see if anyone else has seen anything like this.
Paolo Di Tommaso
@pditommaso
Nov 07 2017 20:10
keep me posted
Mike Smoot
@mes5k
Nov 07 2017 20:11
Will do. I may just see about upgrading the version of slurm I'm using...