These are chat archives for nextflow-io/nextflow

19th
Sep 2016
Evan Floden
@evanfloden
Sep 19 2016 08:57
Thanks Paolo, I had run into the same problems in May when I first tried this. Looks like people are working on it so with any luck we will have NF working on Windows soon.
Paolo Di Tommaso
@pditommaso
Sep 19 2016 09:11
:+1:
Mike Smoot
@mes5k
Sep 19 2016 21:44

@pditommaso Hi Paolo, so I'm still working through the problem we were chatting about late Friday and I think I've found the culprit, although I don't fully understand why this is a problem. Basically, if the input symlinks share the same hash prefix as the current working directory, then the symlinks don't resolve when the container runs. Here's an example:

Current dir is /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/fe/1108fdf113d3d4f71d9bf45febcc99 and the symlinks are as follows:

lrwxrwxrwx. 1 msmoot employees 150 Sep 15 16:32 A0A0B6YBN9.fa -> /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/fe/df9e35bd45aece010c96e000633bd2/genewise_inputs/we03730_nuc_v150325_PM_chr_1/A0A0B6YBN9.fa
lrwxrwxrwx. 1 msmoot employees 151 Sep 15 16:32 we03730_nuc_v150325_PM_chr_1_1505000-1506500-A0A0B6YBN9.gff -> /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/fe/df9e35bd45aece010c96e000633bd2/we03730_nuc_v150325_PM_chr_1_1505000-1506500-A0A0B6YBN9.gff
-rw-r--r--. 1 root   root        0 Sep 15 16:33 we03730_nuc_v150325_PM_chr_1_1505000-1506500-A0A0B6YBN9.gff.fa
lrwxrwxrwx. 1 msmoot employees 184 Sep 15 16:32 we03730_nuc_v150325_PM_chr_1_1505000-1506500.fa -> /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/fe/df9e35bd45aece010c96e000633bd2/genewise_inputs/we03730_nuc_v150325_PM_chr_1/we03730_nuc_v150325_PM_chr_1_1505000-1506500.fa

The important thing to notice is that both the current working dir and directory where the inputs actually live are both in the /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/fe directory.

Now looking at .command.run, I see that the docker mounts are set up as: -v /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/fe:/local_scratch/msmoot/nextflow_eukaryotic_annotation/work/fe -v "$PWD":"$PWD" -w "$PWD" which appears to be the nearest common directory of the current dir and all the inputs.

When I execute bash .command.run it will fail. If I edit the mount points in .command.run to omit the "/fe" prefix like this: -v /local_scratch/msmoot/nextflow_eukaryotic_annotation/work:/local_scratch/msmoot/nextflow_eukaryotic_annotation/work -v "$PWD":"$PWD" -w "$PWD" then the command works.

Any thoughts on why this might be the case? It seems that the mount point shouldn't matter as long as all inputs are a subdirectory of the mount point, but that's not what I'm seeing.

Paolo Di Tommaso
@pditommaso
Sep 19 2016 21:55
ah, so it's related to docker
Mike Smoot
@mes5k
Sep 19 2016 21:55
Like, apparently, most of my problems. :)
Paolo Di Tommaso
@pditommaso
Sep 19 2016 21:56
what version are u using ?
Mike Smoot
@mes5k
Sep 19 2016 21:59
nextflow 0.21.3 and docker 1.11.2
Paolo Di Tommaso
@pditommaso
Sep 19 2016 22:00
um, so it can manage to resolve the realpath of the symlink
Mike Smoot
@mes5k
Sep 19 2016 22:01
No, that seems to be the problem. Because it mounts work/fe instead of work it can no longer resolve the links.
Paolo Di Tommaso
@pditommaso
Sep 19 2016 22:04
can u create an issue attaching the command.run and the list of the directory? I will debug it tomorrow
Mike Smoot
@mes5k
Sep 19 2016 22:06
Sure. I just tried the equivalent of adding result << "-v ${workdir}:${workdir} " in the above code and the command worked. Anyway, I'll enter a ticket with as much info as I have. Strange one.
Paolo Di Tommaso
@pditommaso
Sep 19 2016 22:07
great thanks
there's something that I'm not understanding, the work dir it's always added to the list of the files to mount
so the longest parent should be /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/
not /local_scratch/msmoot/nextflow_eukaryotic_annotation/work/fe
Paolo Di Tommaso
@pditommaso
Sep 19 2016 22:14
sorry too late for me :sleeping:
Mike Smoot
@mes5k
Sep 19 2016 22:15
No worries! I'll keep poking around here to see what I can find and put it in the ticket.