These are chat archives for nextflow-io/nextflow

13th
Feb 2019
Mathias Walzer
@mwalzer
Feb 13 11:51
thanks, good to know
Mathias Walzer
@mwalzer
Feb 13 12:02
@apeltzer @ewels link above does not specify how to use containerOptions in a profile, the naive approach of cluster { docker.enabled = false singularity.enabled = true process.executor = 'slurm' containerOptions = '-B /mnt/gluster:/mnt/gluster' }
didn't work
so how exactly can I add it to my config profile?
Phil Ewels
@ewels
Feb 13 13:05
Nearly there - I think you just need to wrap the above in profile {} and then call it with -profile cluster (as you called it cluster - that can be anything).
If this is going in a config file outside of the pipeline you don’t really need the profile thing though. That’s just to enable easy switching between sets of config vars
If this is for example in your home directory config, you always want it to apply
In which case, just get rid of the outer cluster {} and it should work
Mathias Walzer
@mwalzer
Feb 13 13:13
Oh, yeah my config profile contained several, and I just pasted the 'cluster' entry - the problem why that wasn't working is that containerOptions as is wasn't recognized
I took a guess, and put it under process.containerOptions and this seem to do the trick
thanks
Ola Tarkowska
@olatarkowska
Feb 13 14:16
Hello, I am new to nf and k8s and perhaps have naive question
Paolo Di Tommaso
@pditommaso
Feb 13 14:16
welcome
nobody is supposed to know everything
micans
@micans
Feb 13 14:18
k8s is a fountain of confoundesness
Paolo Di Tommaso
@pditommaso
Feb 13 14:18
I agree, but resists big stuff will arrive soon
or at least I was told so :wink:
micans
@micans
Feb 13 14:19
sounds good!
Alexey Dushen
@blacky0x0
Feb 13 14:19
btw, will NF support CWL (Common Workflow Language) in the future?
Paolo Di Tommaso
@pditommaso
Feb 13 14:19
no
eventually tool descriptions, surely not workflow definition
Ola Tarkowska
@olatarkowska
Feb 13 14:20
So we have upgraded k8s cluster to the latest 2.8 and try to run nextflow run https://github.com/nextflow-io/rnatoy
Anthony Ferrari
@af8
Feb 13 14:20

A really quick thing, I have an input defined as a file which contains a simple string that I want to incorporate in the script command line. Here a snippet to reproduce the example:

file_ch = Channel.fromPath("foo.txt")


process test {
    input: file(myfile) from file_ch

    output: file 'outfile.txt' into out_ch

    script:
    content = myfile.text

    """
    echo ${content} > outfile.txt
    """
}

I got the java.nio.file.ProviderMismatchException on the content = myfile.textline. What am I missing ?

Paolo Di Tommaso
@pditommaso
Feb 13 14:21
@olatarkowska and ?
Ola Tarkowska
@olatarkowska
Feb 13 14:21
distraught-jang:/# nextflow run http://github.com/nextflow-io/rnatoy
N E X T F L O W  ~  version 19.01.0
Launching `nextflow-io/rnatoy` [goofy_kilby] - revision: bfcee3841f [master]
R N A T O Y   P I P E L I N E    
=============================
genome: /workspace/projects/nextflow-io/rnatoy/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa
annot : /workspace/projects/nextflow-io/rnatoy/data/ggal/ggal_1_48850000_49020000.bed.gff
reads : /workspace/projects/nextflow-io/rnatoy/data/ggal/*_{1,2}.fq
outdir: results

[warm up] executor > k8s
[2a/eceed7] Submitted process > buildIndex (ggal_1_48850000_49020000.Ggal71.500bpflank)
ERROR ~ Error executing process > 'buildIndex (ggal_1_48850000_49020000.Ggal71.500bpflank)'

Caused by:
  Process `buildIndex (ggal_1_48850000_49020000.Ggal71.500bpflank)` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

  bowtie2-build --threads 1 ggal_1_48850000_49020000.Ggal71.500bpflank.fa genome.index

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  /bin/bash: .command.run: No such file or directory

Work dir:
  /workspace/at22/work/2a/eceed709cbbaeefb6eefcc0dc2223b

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
Oops .. something went wrong
Paolo Di Tommaso
@pditommaso
Feb 13 14:23
umm, are you running it from a container ?
Ola Tarkowska
@olatarkowska
Feb 13 14:23
from the pod
that is run from /
if I move to my workspace folder /workspace/username/ I am getting cache error
ERROR ~ Can't open cache DB: /workspace/username/.nextflow/cache/c06a65a5-f8a4-49ea-9e40-5df9893df9ef/db

Nextflow needs to be executed in a shared file system that supports file locks.
Alternatively you can run it in a local directory and specify the shared work
directory by using by `-w` command line option.

 -- Check '.nextflow.log' file for details
do you have any thoughts how to debug it?
appology I am new to all of that
Paolo Di Tommaso
@pditommaso
Feb 13 14:27
have you followed the doc ?
Ola Tarkowska
@olatarkowska
Feb 13 14:29
yes I did
Paolo Di Tommaso
@pditommaso
Feb 13 14:29
so how the volume claim is defined ?
Paolo Di Tommaso
@pditommaso
Feb 13 14:31
yes, it must specified in the nextflow.config file
Ola Tarkowska
@olatarkowska
Feb 13 14:31
so I am logging in nextflow kuberun login -c nf.config -v lustre-pvc
Paolo Di Tommaso
@pditommaso
Feb 13 14:31
is there the nextflow.config file ?
Ola Tarkowska
@olatarkowska
Feb 13 14:32
you mean where?
Paolo Di Tommaso
@pditommaso
Feb 13 14:32
once you did kuberun .. etc
in the current dir there must be a nextflow.config file
Ola Tarkowska
@olatarkowska
Feb 13 14:33
# cat nextflow.config 
k8s {
   debug {
      yaml = true
   }
   storageClaimName = 'lustre-pvc'
}

process {
   executor = 'k8s'
}

cacheable = true
runName = 'distraught-jang'
Paolo Di Tommaso
@pditommaso
Feb 13 14:33
so far so good
then you have to run NF in the directory where that file is
or move that file ..
Ola Tarkowska
@olatarkowska
Feb 13 14:34
indeed, but that cause
distraught-jang:/workspace/username# nextflow run https://github.com/nextflow-io/rnatoy
N E X T F L O W  ~  version 19.01.0
Launching `nextflow-io/rnatoy` [deadly_rosalind] - revision: bfcee3841f [master]
ERROR ~ Can't open cache DB: /workspace/username/.nextflow/cache/ef810041-4b8e-4656-a754-b0f2be3caa75/db

Nextflow needs to be executed in a shared file system that supports file locks.
Alternatively you can run it in a local directory and specify the shared work
directory by using by `-w` command line option.

 -- Check '.nextflow.log' file for details
Paolo Di Tommaso
@pditommaso
Feb 13 14:35
well, you have two options
  1. ask your system admins to fix your shared storage so that it supports file locking
  2. launch NF from a different path local to container specifying manually the config file path and the work directory in the shared claim
@af8 file content cannot be accessed in the script prolog ie. content = myfile.text
Anthony Ferrari
@af8
Feb 13 14:40
@pditommaso what's the usual strategy then to have it visible in the script block ?
Ola Tarkowska
@olatarkowska
Feb 13 14:42
@pditommaso so you are saying I incorrectly mounted lustre becuase it doesnt support lock ?
Paolo Di Tommaso
@pditommaso
Feb 13 14:42
so it seems
Ola Tarkowska
@olatarkowska
Feb 13 14:42
yes indeed nextflow -c works
thanks a lot
and sorry for silly question
Paolo Di Tommaso
@pditommaso
Feb 13 14:43
:+1:
@af8
"""
cat $myfile 
"""
Anthony Ferrari
@af8
Feb 13 14:46
yes I have this bit, but how do you store the cat output in a variable ?
Sorry I am literally bugging on this one :-)
Paolo Di Tommaso
@pditommaso
Feb 13 14:47
not possible, you need to pass trough a file
well a NF variable, you can still use a Bash variable in the script context
Anthony Ferrari
@af8
Feb 13 14:48
ahh OK, I can't feed a NF variable
Paolo Di Tommaso
@pditommaso
Feb 13 14:48
nope
Anthony Ferrari
@af8
Feb 13 14:51

So need to go with sth like this

    """
    VAR=\$(cat ${myfile})
    echo \$VAR > outfile.txt
    """

thanks

Paolo Di Tommaso
@pditommaso
Feb 13 14:53
well cat $myfile > outfile.txt does the same !
Anthony Ferrari
@af8
Feb 13 14:54
yeah sure :-) but the echo command was just for the example !
Paolo Di Tommaso
@pditommaso
Feb 13 14:54
ok
Luca Cozzuto
@lucacozzuto
Feb 13 15:51
hi, I got this error when using a singularity image
ERROR  : There was an error remounting /var/singularity/mnt/session/passwd to /var/singularity/mnt/final/etc/passwd: Invalid argument
ABORT  : Retval = 255
I know that is not strictly Nextflow... but I don't know if you experienced the same :) (singularity version 2.5)
Paolo Di Tommaso
@pditommaso
Feb 13 16:06
passwd ?!
Luca Cozzuto
@lucacozzuto
Feb 13 16:06
no idea
I found this
it could be a bug...
Mathias Walzer
@mwalzer
Feb 13 16:43
I still haven't gotten the groovy concept behind all, but maybe this will help me see through: so I have a file in a result channel of a process. Now I want to store all of these in S3. I've seen from example, how I can print each line from files in a channel, but I can't see how to go from there to storing to s3 - can you help me?
Alexander Peltzer
@apeltzer
Feb 13 17:09
Define an output directory on an s3 bucket you have permissions for and the results will be uploaded there ?
Your description is a bit unclear to me tbh
Mathias Walzer
@mwalzer
Feb 13 17:12
yeah, publishDir is an option of course if you mean that???
but I was trying to understand what I can do with files in a channel
Alexander Peltzer
@apeltzer
Feb 13 17:16
Passing around files and values between processes
Mathias Walzer
@mwalzer
Feb 13 17:17
but thats not all I can do with them, or is it?
Alexander Peltzer
@apeltzer
Feb 13 17:17
Your question suggested you want to store the output of something to s3, that’s why I recommended publishDir :+1:
Mathias Walzer
@mwalzer
Feb 13 17:18
like for example this results.collectFile(name: params.out).println { file -> "Whats in the files:\n ${file.text}" }
where results is a channel
and that bit of code can be just after all the processes