These are chat archives for nextflow-io/nextflow

7th
May 2019
Rad Suchecki
@rsuchecki
May 07 00:13
You can also:
  • Use globs e.g. file 'sample.vcf*' into variants_ch
  • Put both files into a subdir and use file subdir into variants_ch (a bit clunky)
    @olavurmortensen
Ólavur Mortensen
@olavurmortensen
May 07 08:20
@rsuchecki For some reason using file "sample.vcf*" into variants_ch gives me input files sample.vcf1 and sample.vcf.idx1, which is why I didn't go for using glob, although that is much more clean.
micans
@micans
May 07 12:46
@olavurmortensen I don't understand the problem yet. A tool doesn't understand symbolic links, and the way in you can solve it is cumbersome to write, but I haven't quite connected the two yet.
Ólavur Mortensen
@olavurmortensen
May 07 12:49
@micans To put it differently, I want to include the index file as output, written in a concise way. I like what @rsuchecki suggested, file "sample.vcf*" into variants_ch, but as I mentioned above, I can't quite make it work.

I have one way that works. First the process that outputs the channel states:

output:
file "sample.vcf*" into variants_ch

And then the process that takes the channel as input states:

input:
file vcf from variants_ch
micans
@micans
May 07 12:54
@olavurmortensen ok, that looks good to me and standard.
Your remark about symbolic link issues caused me to search for staging files using hard links: https://www.nextflow.io/docs/latest/process.html#stageinmode . Not sure it's relevant.
Ólavur Mortensen
@olavurmortensen
May 07 12:57
Good to know. Although 90% this is no problem, I just output the file itself and the tool finds the index file when it follows the symbolic link.
Ólavur Mortensen
@olavurmortensen
May 07 13:10
Correction: that doesn't work because in this case the value of vcf will be sample.vcf sample.vcf.idx.
micans
@micans
May 07 13:15
@olavurmortensen then you need vcf[0] and vcf[1] -- vcf is an array of files.
Although best not to use the glob with that, and be more specific in enumerating the two files.
Ólavur Mortensen
@olavurmortensen
May 07 13:26

Although my original syntax is a bit cumbersome it seems to be the best solution. That is, in one process:

output:
set file('sample.vcf'), file('sample.vcf.idx') into variants_ch

And then the subsequent process:

input:
set file(vcf), file(idx) from variants_ch
micans
@micans
May 07 13:28
It looks good to me. Clear and concise, I would not describe it as cumbersome. Think of the person needing to maintain your code, it might be you.
Ólavur Mortensen
@olavurmortensen
May 07 13:33
haha yeah exactly
Anthony Underwood
@aunderwo
May 07 19:46

Hi all. I've got a weird problem running Nextflow on MacOS X. Previously this was not a problem

I get the error

  .command.stub: line 84: cannot create temp file for here-document: No space left on device

However df -h says

Filesystem      Size   Used  Avail Capacity     iused               ifree %iused  Mounted on
/dev/disk1s1   466Gi  352Gi  111Gi    77%     2482701 9223372036852293106    0%   /
devfs          231Ki  231Ki    0Bi   100%         798                   0  100%   /dev
/dev/disk1s4   466Gi  2.0Gi  111Gi     2%           2 9223372036854775805    0%   /private/var/vm
map -hosts       0Bi    0Bi    0Bi   100%           0                   0  100%   /net
map auto_home    0Bi    0Bi    0Bi   100%           0                   0  100%   /home
drivefs        466Gi  358Gi  108Gi    77% 18446744069414590987          4294967295 276728833924611296%   /Volumes/GoogleDrive

root partition has space and I can create large files on /tmp no problem. I've asked around the office and I've drawn a blank so wondered if the 'hive mind' might be able to help

Anthony Underwood
@aunderwo
May 07 19:57
line 84 is
IFS=' ' read -a val <<< "$nxf_pstat_ret"; unset IFS
Alexander Peltzer
@apeltzer
May 07 20:17
Are you using Docker?
I ran into that using the Docker For Mac Application - once the 64GB default image space in that application is full, you'll see these kinds of errors
Anthony Underwood
@aunderwo
May 07 20:19
Yes I am. Should I clean up dangling images?
Alexander Peltzer
@apeltzer
May 07 20:19
Yes
If you click on the Whale Icon -> Preferences -> Disk
It should show how much is used up
Anthony Underwood
@aunderwo
May 07 20:20
image.png
I just see
image.png
In finder it shows the file size as 64Gb
Alexander Peltzer
@apeltzer
May 07 20:22
When I had these errors, it was just running full
getting rid of some old containers I don't need was enough to get this settled
Otherwise, I don't have an idea :-(
Anthony Underwood
@aunderwo
May 07 20:23
However I have just cleaned up dangling images - not many, around 8 and it's working!!!! I shall look into what I need to do to increase capacity. Thanks @apeltzer - the community is fab. Hive mind FTW!!!
Alexander Peltzer
@apeltzer
May 07 20:32
No prob, ran into this myself already - happy to help