These are chat archives for nextflow-io/nextflow

5th
Mar 2019
Laurence E. Bernstein
@lebernstein
Mar 05 00:46
@stevekm I was not useing the overwrite flag only the mode:copy so my results were as expected. Not recopying.

Meanwhile.. I have another frustration. I am attempting (as I mentioned in a previous post) to run nextflow from inside of nextflow, which seems to be working, however I want to specify my configuration file "on-the-fly". Due to what I think is the bug that people were encountering with configuration files not in the same directory I want/need to stage the file so it will be found in the current directory when nextflow is executed. Therefore I want to do something like:

input:
val projectName from project_name_ch.splitText()
file workflow from file("${projectName}")

But this does not work. I don't like this method really, but is there another way? Or am I doing something wrong?

Riccardo Giannico
@giannicorik_twitter
Mar 05 10:33
Hi guys, docker with autofs related question:
I have a .nf running one of the processes in a docker, so I need to mount some volumes inside docker.
Problem is I have a /storage folder with a lot of 'volume1,volume2,volume3' folders mounted with autofs.
How can I mount directly my /storage in docker being sure all autofs volumes are accessible?
Do I have to install autofs inside the docker? Or is there some more elegant option with nextflow?
Riccardo Giannico
@giannicorik_twitter
Mar 05 10:52
My problem is that simply mounting /storage -v /storage:/storage and accessing some volumes from the container I have a "Too many levels of symbolic links" error.
I found I can temporarily solve it (I don't know why) simply restarting the docker service sudo service docker stop ; sudo service docker startbut the issues returns after a while. I suppose it only works as long as the linux launching the docker has the volume mounted by autofs, and then stops working when his autofs unmounts the volume. But of course I have no control on it from the docker.
Paolo Di Tommaso
@pditommaso
Mar 05 11:44
umm, autofs and docker doesn't work well
take in consideration to use singularity instead
Riccardo Giannico
@giannicorik_twitter
Mar 05 12:10
ok, thanks for the tip :)
Luca Cozzuto
@lucacozzuto
Mar 05 12:31

Hi all, I have a channel formatted as this:

[chr6.bed, -V ERR2433452.g.vcf.gz -V ERR2433491.g.vcf.gz, [/myfilepath/ERR2433452.g.vcf.gz, /myfilepath/ERR2433491.g.vcf.gz, /myfilepath/ERR2433452.g.vcf.gz.tbi]

And I got an error when try to use this process

process GenomicsDBImport {

    input:
    set val(interval), val(pars), file("*") from gvcf_per_interval

    script:   
    """
    echo ${pars}
    """
}
the error:
ERROR ~ Error executing process > 'GenomicsDBImport (1)'

Caused by:
  java.nio.file.ProviderMismatchException
Paolo Di Tommaso
@pditommaso
Mar 05 12:32
-V ... 🤦
Luca Cozzuto
@lucacozzuto
Mar 05 12:32
Is this the problem?
Canno force a string?
Paolo Di Tommaso
@pditommaso
Mar 05 12:33
it could
Luca Cozzuto
@lucacozzuto
Mar 05 12:35
not working even if I remove the -V
I replaced the text with a single word
Paolo Di Tommaso
@pditommaso
Mar 05 12:36
🤷
I have an entire catalog of emojis :D
Luca Cozzuto
@lucacozzuto
Mar 05 12:37
thanks
so now my channel is
[chr6.bed, mytext, [/myfilepath/ERR2433452.g.vcf.gz, /myfilepath/ERR2433491.g.vcf.gz, /myfilepath/ERR2433452.g.vcf.gz.tbi]]
but this is not capturing
set val(interval), val(pars), file("*") from gvcf_per_interval
Paolo Di Tommaso
@pditommaso
Mar 05 12:38
how is created ?
Luca Cozzuto
@lucacozzuto
Mar 05 12:39
the channel?
Paolo Di Tommaso
@pditommaso
Mar 05 12:39
yes
Luca Cozzuto
@lucacozzuto
Mar 05 12:39
gvcf_per_interval = gvcf_files.groupTuple().map{
        def interval = it[0]
        def pars = "ciccio"
        def files = it[1] + it[2]
        [interval, pars, files]
}
mytext = ciccio :)
Paolo Di Tommaso
@pditommaso
Mar 05 12:41
and what is supposed to be def files = it[1] + it[2] ?
well, let's go directly to the next step
since I know my chicken I guess you meant
def files = [ it[1],  it[2] ]
Luca Cozzuto
@lucacozzuto
Mar 05 12:42
ok
chickens ;)
Paolo Di Tommaso
@pditommaso
Mar 05 12:45
yeah.. more than one :satisfied:
Luca Cozzuto
@lucacozzuto
Mar 05 12:45
btw def files = [ it[1], it[2] ]
this gives me errors
as well
:) well I'll wait for you for an "interested" coffee later ;)
Paolo Di Tommaso
@pditommaso
Mar 05 12:48
the emoji for the holy water has not yet invented unfortunately
Luca Cozzuto
@lucacozzuto
Mar 05 12:48
well this helps a lot
;)
the problem is why is complaining for a java.nio.file.ProviderMismatchException
the channel is composed by 3 elements
2 texts and one list
Alexander Peltzer
@apeltzer
Mar 05 13:21
Is there a way to raise an error and break if any "unknown" parameters are set? atm, its always possible to have --bla and without explictily checking for the existence of --blaI cannot be sure that users supply non-existent stuff or am I wrong?
Not that I really need this, its more of a question ;-)
Paolo Di Tommaso
@pditommaso
Mar 05 13:24
nope at this time
Luca Cozzuto
@lucacozzuto
Mar 05 15:35
Another question... I have something like 2 million jobs... so NF crashed... there is a way to group the jobs in arrays?
Riccardo Giannico
@giannicorik_twitter
Mar 05 16:01
@pditommaso , sorry another question about docker and mounting ..
From nextflow docs I see this warning "Nextflow automatically manages the file system mounts each time a container is launched depending on the process input files. Note, however, that when a process input is a symbolic link file, the linked file must be stored in the same folder where the symlink is located, or any its sub-folder. Otherwise the process execution will fail because the launched container won't be able to access the linked file."
Can you please help me to better understand what NF does automatically mounts and what it doesn't?
For example if I run a .nf where process1 does NOT use docker but process2 does. Can process2 inside the docker see files from process1 ? Or I have to mount the volume where process1 wrote the "work" directory ?
Jonathan Manning
@pinin4fjords
Mar 05 16:11
Hello. I'm trying to use .watchPath() to pick up new files in a directory. Works fine from the head node, but as soon as I try to run the 'watcher' from a worker node it fails to pick up new files. Something to do with shared filesystems? Any ideas/ workarounds?
Jonathan Manning
@pinin4fjords
Mar 05 16:26
... also if I SSH to the worker node and update the files there, it does trigger the watcher
Luca Cozzuto
@lucacozzuto
Mar 05 16:30
@pinin4fjords careful that the worker node could not be able to submit jobs in other nodes...
Jonathan Manning
@pinin4fjords
Mar 05 16:31
Thanks @lucacozzuto , but I know that it can
Daniel E Cook
@danielecook
Mar 05 17:01
This message was deleted

Can anyone provide some insight into why caching was dropped on a setup that looks like this:

https://gist.github.com/danielecook/e9d60d606e6739da7eef176efe954272

I started with 3-4 methods, then once I saw some had issues I dropped them and added a few more. When I go to run it, however, the cache appears to have been dropped and everything is rerun from scratch
Is there a way to provide more output regarding how the cache is determined and does anyone have any idea why the cache would have been dropped for existing things that were run?
Laurence E. Bernstein
@lebernstein
Mar 05 17:28

Can I do something like this?

 input:
    val projectName from project_name_ch.splitText()
    file workflow from file("${projectName}")

I want to get the name of a file from another file and make sure it is staged by nextflow so that I will have access to it.
Also this doesn't seem to work either:

 input:
    val projectName from project_name_ch.splitText()
  script:
    if (projectName == "XXX") {

It evaluates to false even though "XXX" is what is in the file.

Félix C. Morency
@fmorency
Mar 05 18:28
I recall some discussion about the ability for NF to "merge" what would otherwise be lots of small processes into one "bigger" process that would be executed on a node. I can't seem to find the issue about it. Any hint?
Félix C. Morency
@fmorency
Mar 05 18:35
Mmmm I might be able to perform this using each
Can each contain a set?
Tobias "Tobi" Schraink
@tobsecret
Mar 05 18:37

@lucacozzuto

Another question... I have something like 2 million jobs... so NF crashed... there is a way to group the jobs in arrays?

You can limit how many jobs you are submitting per second and how many jobs can be in queue at the same time.

executor {
  submitRateLimit = '<limit>'
  queueSize='<limit>'
}
Félix C. Morency
@fmorency
Mar 05 18:38
Yeah I don't think each will do the trick