These are chat archives for nextflow-io/nextflow

16th
Jun 2017
Paolo Di Tommaso
@pditommaso
Jun 16 2017 09:05
@bioinfo-geekz You can reduce the amount of storage by setting process.scratch = true in your config file
by doing that tasks are computed in the node local storage, tho intermediate output files will still copied in the cluster shared storage
LukeGoodsell
@LukeGoodsell
Jun 16 2017 09:21
Hey Paolo, is there any plan to implement #296 (optionally use relative paths in symlinks)? This is increasingly become a pain for my group working on shared folder more moving projects (rendering the symlinks invalid), as well as accessing data within docker instances.
LukeGoodsell
@LukeGoodsell
Jun 16 2017 09:53
We currently litter our processes with thinks like this as a workaround:
        if [[ -h !{file} ]]; then
            relpath=$(realpath --relative-to=./ !{file});
            rm !{file};
            ln -s ${relpath} !{file};
        fi
Paolo Di Tommaso
@pditommaso
Jun 16 2017 10:49
@LukeGoodsell can't promise, but I may have some time in july to give a try to this
LukeGoodsell
@LukeGoodsell
Jun 16 2017 10:50
Great! I’d really appreciate it
Félix C. Morency
@fmorency
Jun 16 2017 13:44
+1 ... We're using SMB multiuser feature so it's less of a big deal, but still, it would be nice
Evan Floden
@evanfloden
Jun 16 2017 15:57
Is it true that when I run via nextflow run <organisation>/<repo> if the repo exisits in .nextflow/assests then any changes in the repo will no be pulled. I.e., NF runs from the local copy and does not check for any updates in the repo. And if true, is this what users would expect?
Félix C. Morency
@fmorency
Jun 16 2017 15:57
iirc, yes it is true
Phil Ewels
@ewels
Jun 16 2017 16:04
It is true (you need to do nextflow pull to get the latest copy) and in my experience people are usually surprised by this :) (though it's good behaviour for reproducibility)
A warning if the remote has changed would be really good actually :+1:
Evan Floden
@evanfloden
Jun 16 2017 16:08
Thanks @fmorency + @ewels. I completely agree that a warning would be ideal. Will add as feature request.
Paolo Di Tommaso
@pditommaso
Jun 16 2017 16:12
though it's good behaviour for reproducibility
that's the point
you cannot quietly update a local user pipeline
Evan Floden
@evanfloden
Jun 16 2017 16:17
For sure, but if I run a pipeline once, then 6 months later, I foget I have run it (very possible with me anyway) then at least I should get a message that I have done this.
Paolo Di Tommaso
@pditommaso
Jun 16 2017 16:19
um, that would require to make a request to GH each time you launch
there's the option -latest for that
Phil Ewels
@ewels
Jun 16 2017 16:31
Nice, didn't know about -latest
Maxime Garcia
@MaxUlysse
Jun 16 2017 16:31
Unless you're on a cluster without internet access ;-)
Phil Ewels
@ewels
Jun 16 2017 16:31
It's easy to try and fail quickly :)
Maybe just a log info message saying that it's using a cached version of the pipeline?
And perhaps a date from when the cached version was fetched?
Félix C. Morency
@fmorency
Jun 16 2017 16:36
You can already print some workflow/git info https://www.nextflow.io/docs/latest/metadata.html
It's not exactly what you asked but it's better than nothing.
Paolo Di Tommaso
@pditommaso
Jun 16 2017 16:39
When available the git commit is printed when the run starts
Now sorry guys :sailboat:
:)
Félix C. Morency
@fmorency
Jun 16 2017 16:42
Have a nice one @pditommaso