These are chat archives for nextflow-io/nextflow

23rd
Apr 2019
Ólafur Haukur Flygenring
@olifly
Apr 23 12:40

Hi.

I'm currently having problem configuring the K8s exectutor while running nextflow version 19.04.0

My config for it is defined as such:

k8s {
storageClaimName = 'pvc-nfs'
storageMountPath = '/mnt/abc'
namespace = 'devel'
projectDir = '/mnt/abc/workflow/project'
}

The problem is that the executor seems to ignore the project dir, ie it doesn't download the project to this location, but instead uses the /.nextflow/assets directory within the pod that's running the nextflow executable.

This results in none of the processes executing having access to files included within the repository (for example the /bin directory).

Am I missing some crucial config information or is this a bug? :)

Best Regards,
Ólafur Haukur Flygenring

Paolo Di Tommaso
@pditommaso
Apr 23 12:42
can you find this entry in your log ?
Ólafur Haukur Flygenring
@olifly
Apr 23 12:46
Not seeing it no :(
Paolo Di Tommaso
@pditommaso
Apr 23 12:47
it may be a bug ..
Ólafur Haukur Flygenring
@olifly
Apr 23 12:50
I'm not using the kuberun command. I'm using the normal run command with a script and a config that specifies the k8s executor
Paolo Di Tommaso
@pditommaso
Apr 23 12:51
ah, if so k8s config context is not used (I guess)
the usual NF logic applies in that case
Ólafur Haukur Flygenring
@olifly
Apr 23 12:52
hmmm, but it's using the other config parameters from it, namely the storageClaimName and storageMountPath
and the namespace
but not the projectDir
Paolo Di Tommaso
@pditommaso
Apr 23 12:54
umm .. you are right
and here
therefore projectDir is not used in the scenario
Ólafur Haukur Flygenring
@olifly
Apr 23 12:56
Is that intended behaviour?
Paolo Di Tommaso
@pditommaso
Apr 23 12:57
yes
Ólafur Haukur Flygenring
@olifly
Apr 23 12:58
Ok, but if it is ignored, then scripts being run with the k8 executor via the run command won't have access to files included in the script repository
Paolo Di Tommaso
@pditommaso
Apr 23 12:58
if you need to control there projects are downloaded use usual NF vars ie. NXF_ASSETS
Ólafur Haukur Flygenring
@olifly
Apr 23 12:58
for example the /bin directory
Ólafur Haukur Flygenring
@olifly
Apr 23 12:59
Ok, the documentation needs updating then to clarify that projectDir is only used with kuberun and not run I would think
Paolo Di Tommaso
@pditommaso
Apr 23 12:59
then scripts being run with the k8 executor via the run command won't have access to files included in the script repository
this should be unrelated to projectDir
Ólafur Haukur Flygenring
@olifly
Apr 23 13:02

The documentation on projectDir suggests that for the k8s executor the projectDir config entry is doing similar things to NXF_ASSETS and makes no mentions of it only working when using the kuberun command :)

"Defines the path where Nextflow projects are downloaded. This must be a path in a shared K8s persistent volume "

But ok, I'll switch to using the NXF_ASSETS env var instead :)
Paolo Di Tommaso
@pditommaso
Apr 23 13:03
we do our best .. :)
Ólafur Haukur Flygenring
@olifly
Apr 23 13:03
Thanks for the clarification :)
Paolo Di Tommaso
@pditommaso
Apr 23 13:03
welcome
Chelsea Sawyer
@csawye01
Apr 23 13:24
Is there a way to record the errors that occur to an output file if the errorStrategy for a process is set to ignore?
Paolo Di Tommaso
@pditommaso
Apr 23 13:27
nextflow run <script> -with-trace
than look at the trace.txt file
Chelsea Sawyer
@csawye01
Apr 23 13:30
Thank you!
Paolo Di Tommaso
@pditommaso
Apr 23 13:30
welcome
Vladimir Kiselev
@wikiselev
Apr 23 14:35
hi all! is it possible to check what container NF is running in? I can’t find a container definition in any of the reports? Am I missing something?
Paolo Di Tommaso
@pditommaso
Apr 23 14:37
Screenshot 2019-04-23 at 16.36.45.png
there's in my computer :D
Vladimir Kiselev
@wikiselev
Apr 23 14:41
ok, good-good, what if I didn’t create a report file? ;-)
Paolo Di Tommaso
@pditommaso
Apr 23 14:42
you can check the execution config with nextflow config in the launch path
Vladimir Kiselev
@wikiselev
Apr 23 14:47
oh, nice, many thanks @pditommaso !!
Paolo Di Tommaso
@pditommaso
Apr 23 14:50
:v:
Vladimir Kiselev
@wikiselev
Apr 23 15:16
@pditommaso, does this only work if there is a nextflow.config file in the launch directory?
Paolo Di Tommaso
@pditommaso
Apr 23 15:17
it works with the same logic of run
therefore you can specify also the script/project as in the run command line
Vladimir Kiselev
@wikiselev
Apr 23 15:20
oh, I see, nice, now all is clear
thanks!
Paolo Di Tommaso
@pditommaso
Apr 23 15:20
welcome
Thomas Zichner
@zichner
Apr 23 15:20
Hi @pditommaso and others,
When using Singularity, is there a way to always run a specific command before NF does singularity pull ...? The reason is that we need to dynamically set some singularity authentication variables.
Thanks
Similary to process.beforeScript just with the difference that it is executed before the pipeline starts rather than before a process starts
Paolo Di Tommaso
@pditommaso
Apr 23 16:49
nope, at this time there isn't such mechanism