These are chat archives for nextflow-io/nextflow

19th
Nov 2018
Krittin Phornsiricharoenphant
@sinonkt
Nov 19 2018 03:48
@rsuchecki thank you so much. :))
Paolo Di Tommaso
@pditommaso
Nov 19 2018 06:48
@sinonkt use the quiet option nextflow -q run sinonkt/test-nf
Excellent that minio works fine !
@Fizol I would love, so far there's only grammar for Atom and VS Code. I guess Intellj uses a complete different parsing approach.
@tobsecret hard to say without more details
@kerimoff It looks like a problem with singularity file system mounts, I would suggest to address this question in the nf-core channel
Tim Dudgeon
@tdudgeon
Nov 19 2018 15:15
I have a Q about singularity. If you use singularity pull ... to pull and convert a docker image you end up with an image with a .sif extension.
If however you let nextflow do the pulling you end up with an image with a .img extension and an identical .sif file in the $NXF_SINGULARITY_CACHEDIR dir is ignored. Is there a way around this?
Paolo Di Tommaso
@pditommaso
Nov 19 2018 15:17
with an image with a .img extension and an identical .sif
sound strange, it should only create .img
Tim Dudgeon
@tdudgeon
Nov 19 2018 15:31
$ singularity pull docker://busybox
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:90e01955edcd85dac7985b72a8374545eac617ccdddcc992b732e43cd42534af
 710.92 KiB / 710.92 KiB [==================================================] 0s
Copying config sha256:d98834fba17e4121dc21f65d6ddf2f648da119c86d72ffea8145e496bda621fd
 574 B / 574 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: busybox_latest.sif
Paolo Di Tommaso
@pditommaso
Nov 19 2018 15:33
yes, I was meaning the image produced by NF
it simply uses the --name option, that gives to the pulled image an alternate name
Tim Dudgeon
@tdudgeon
Nov 19 2018 15:39
Sorry I was not clear. Nextflow creates a .img file even in a .sif file is already present. NF does not create both.
Max Hargreaves
@DiDeoxy
Nov 19 2018 16:07
Hi, I am having a problem running nextflow on the Compute Canada system Cedar. I am trying to submit to the SLURM queue from within an interactive session (so the Java VM can run, restricted on login nodes) both sbatch and singularity are accessible within the session (I am using images which are downloaded and available). However when I run nextflow I get the following output:
N E X T F L O W  ~  version 18.10.1
Launching `main.nf` [determined_snyder] - revision: 0b45bffdc0
run.sh: line 1: 168437 Killed                  nextflow -C cedar.config run -resume main.nf -with-report pipeline_report.html
Paolo Di Tommaso
@pditommaso
Nov 19 2018 16:18
run.sh: line 1: 168437 Killed ¯\(ツ)
@tdudgeon, that's expected NF checks only for .img file, just rename .sif to .img
tbugfinder
@tbugfinder
Nov 19 2018 18:51
Hello, I have a list of S3 input files. As larger files take more time to complete I'd like to feed larger files first into AWS Batch. toSortedList and flatten doesn't eject single items to the process step?
Channel
  .fromPath(mys3)
  .map { file -> tuple (file, file.size()) }
  .view()
  .set { iC3 }
Paolo Di Tommaso
@pditommaso
Nov 19 2018 18:54
do you mean ?
Channel.from(1,2,3)
    .toSortedList()
    .flatten()
    .println()
1
2
3
tbugfinder
@tbugfinder
Nov 19 2018 19:13
Sadly, not
Input is a list of files and it should be sorted by its size.
Tobias "Tobi" Schraink
@tobsecret
Nov 19 2018 19:15
@pditommaso : could the NextFlow executor being killed have something to do with my cluster's ulimit settings?
[user@host]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515184
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
tbugfinder
@tbugfinder
Nov 19 2018 19:17
@tobsecret Did you check message log ?
Tobias "Tobi" Schraink
@tobsecret
Nov 19 2018 19:22
yes, it just cuts off abruptly and the executor gets killed.
The last bit of console output usually looks something like this:
[c5/58e36a] Submitted process > align_reads_bwa (SampleID ERS347585 SampleSubID ERR216611)
Killed
Paolo Di Tommaso
@pditommaso
Nov 19 2018 19:24
are you running in a batch scheduler env ?
Tobias "Tobi" Schraink
@tobsecret
Nov 19 2018 19:26
Do you mean what I have in my nextflow.config?
executor {
    name = 'slurm'
    queueSize = 200
}
Paolo Di Tommaso
@pditommaso
Nov 19 2018 19:27
nope, no I mean the env in which the NF driver app is running
is the cluster head node?
Tobias "Tobi" Schraink
@tobsecret
Nov 19 2018 19:28
yep
The cluster manager told me it's fine because it's not using too much CPU
Paolo Di Tommaso
@pditommaso
Nov 19 2018 19:29
if it were crashing because mem is not enough there would be a OutOfMemException in the log
usually this happens because there's some script in the head node killing process taking too much mem
double check with your sysadmin
Tobias "Tobi" Schraink
@tobsecret
Nov 19 2018 19:30
mhmm, does using storeDir tend to increase the memory footprint of the executor?
Paolo Di Tommaso
@pditommaso
Nov 19 2018 19:32
it should not, don't forget the java gc eat as much mem as it can
try to set min-max heap size
Tobias "Tobi" Schraink
@tobsecret
Nov 19 2018 19:48
so nextflow run pipeline.nf -resume -Xms512M -Xmx4G?
Paolo Di Tommaso
@pditommaso
Nov 19 2018 19:52
nope
 NXF_OPTS='-Xms512M -Xmx4G' nextflow run <etc>
Tobias "Tobi" Schraink
@tobsecret
Nov 19 2018 20:05
aaaah, thanks!
tbugfinder
@tbugfinder
Nov 19 2018 20:06

I'm close to what I'm looking for:

Channel
        .fromPath(params.s3dir)
        .map { file -> tuple (file, file.size()) }
        .set { iCPrep }

Channel
        .from iCPrep
        .toSortedList({ a, b -> b[1] <=> a[1] })
        //.flatten()
        //.set { inputforprocess }
        //.view()
        .println()

.flatten() splits any item, but I'd need (file,filesize) as output.

Tobias "Tobi" Schraink
@tobsecret
Nov 19 2018 20:42
Update: seems to be working for the moment, will monitor closely, thanks for your help @pditommaso
Sergey Venev
@sergpolly
Nov 19 2018 23:06
Hi ! I'm trying to understand singularity-containers usage with nextflow ...
I can successfully mount paths to our container singularity shell --bind /cluster:/image:rw img, so that ls /image gives me whatever is in /cluster...
But pipelines crash, when I use singularity.bindOptions="--bind /cluster:/image:rw", with the error message: "file /cluster/blah-blah-file does not exist" - so clearly nextflow tries to refer to that file as /cluster/blah-blah-file, not as /image/blah-blah-file ...
Is this intentional ? Does this part of the docs: https://www.nextflow.io/docs/latest/singularity.html#images confirms what I see - I couldn't really understand the "system-wide" wording?
Our HPC admins told us that our cluster does not support overlayFS or something like that, so that we cannot enjoy the auto mounting feature of singularity ...
Linux ghpcc06 2.6.32-754.6.3.el6.x86_64 #1 SMP Tue Sep 18 10:29:08 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Sergey Venev
@sergpolly
Nov 19 2018 23:12
they said our kernel is too old ... Does anyone have an experience with that - is 2.6.32 indeed too old for that ?