These are chat archives for nextflow-io/nextflow

6th
Sep 2017
spaceturtle
@spaceturtle
Sep 06 2017 07:43
How to enforce a nextflow process to use local storage on sge nodes, rather than central storage of sge system?
Paolo Di Tommaso
@pditommaso
Sep 06 2017 07:43
add process.scratch = true in the config file
spaceturtle
@spaceturtle
Sep 06 2017 07:46
Thanks
Paolo Di Tommaso
@pditommaso
Sep 06 2017 07:46
welcome
Emilio Palumbo
@emi80
Sep 06 2017 08:35
@pkaleta adding the image name to the cache sounds good to me too. I think it make sense also for Singularity.
Anthony Underwood
@aunderwo
Sep 06 2017 09:25
Hi all - has anybody got experience of starting nextflow for 100s of samples, leaving it running and then querying progress from another place?
Evan Floden
@evanfloden
Sep 06 2017 09:25
@aunderwo My favourite topic at the moment!
@pditommaso If the html was rendered on the fly, we could set the output to a public_html location on a filesystem. Something like:
nextflow run <pipeline name> -with-timeline ~/my_public_html/pipeline_name.html
Paolo Di Tommaso
@pditommaso
Sep 06 2017 09:31
to the same with the -with-trace
a bit rustic but works
Evan Floden
@evanfloden
Sep 06 2017 09:31
Is the html timeline rendered in “real-time” during execution?
Paolo Di Tommaso
@pditommaso
Sep 06 2017 09:32
the problem is not NF, is to render the HTML that has a nested structure
you can only render when you have the complete document
tho some workarounds are surely possible
Anthony Underwood
@aunderwo
Sep 06 2017 09:34
OK so trace is dynamic?? We're thinking of trying to build an API to submit Nextflow jobs and then query progress
Paolo Di Tommaso
@pditommaso
Sep 06 2017 09:35
yep
Anthony Underwood
@aunderwo
Sep 06 2017 09:35
That could work then
Will report back if we make any progress
Paolo Di Tommaso
@pditommaso
Sep 06 2017 09:37
:+1:
Anthony Underwood
@aunderwo
Sep 06 2017 09:38
Won't be for a little while but definitely a priority for us if we want to use Nextflow for processing of routine public health/clinical pathogen samples
Phil Ewels
@ewels
Sep 06 2017 09:57
I've been thinking of writing a web front-end for visualising trace files too. Not quite the same, but similar. I might look at this at the upcoming nextflow-io/hack17#4 if we have time..
(Link was meant to say "hackathon")
Simone Baffelli
@baffelli
Sep 06 2017 13:16
Is it possible for exec in a process to produce a "file" output?
I'm trying, but nextflow complains about missing files
Paolo Di Tommaso
@pditommaso
Sep 06 2017 13:17
yes but you need to specify the working folder using task.workDir
Simone Baffelli
@baffelli
Sep 06 2017 13:17
in the file path?
Paolo Di Tommaso
@pditommaso
Sep 06 2017 13:18
no, when you create them
see #378
Simone Baffelli
@baffelli
Sep 06 2017 13:20
how? I don't see any reference to workDir
ok found it out
Simone Baffelli
@baffelli
Sep 06 2017 14:06
Is there a way to keep only one header when using "collectFile"? If not, would make it sense to modify it in order to support this feature? I'm thinking about the case where several csv files are concatenated.
of course there is the skip parameter, but it would be nice to keep at least one header
Paolo Di Tommaso
@pditommaso
Sep 06 2017 14:08
Simone Baffelli
@baffelli
Sep 06 2017 14:08
yes, but I'm thinking something that would re-append the first skipped line after having collected all files
in order to have a long table with the original header
instead of having "header;data1;header;data2" etc, one would like to have "header;data1;data2"
Luca Cozzuto
@lucacozzuto
Sep 06 2017 14:10
+1
it is quite common to have this issue after parallelization
Paolo Di Tommaso
@pditommaso
Sep 06 2017 14:11
I think seed allow you to do that
a bit cryptic name but it allows to prepend an header to the result file
Luca Cozzuto
@lucacozzuto
Sep 06 2017 14:12
would be nice to have an example
Simone Baffelli
@baffelli
Sep 06 2017 14:13
.collectFile(skip: 1, seed: {it->it.readLine()})
Let's try that
Paolo Di Tommaso
@pditommaso
Sep 06 2017 14:14
almost
Simone Baffelli
@baffelli
Sep 06 2017 14:14
just a wild guess
Paolo Di Tommaso
@pditommaso
Sep 06 2017 14:14
it accepts only static value, no closure
Simone Baffelli
@baffelli
Sep 06 2017 14:14
but I want the seed from the file :(
Prepare to accept a PR ;)
Paolo Di Tommaso
@pditommaso
Sep 06 2017 14:16
oops
Simone Baffelli
@baffelli
Sep 06 2017 14:31
for the time being, I will just keep them and remove the redunant headers using dplyr
Mike Smoot
@mes5k
Sep 06 2017 15:37
@aunderwo @ewels @skptic I've got a branch of nextflow somewhere that creates a SocketIO (websockets) server which emits trace events. I've hooked up very simple web pages to the output, but I've never had the patience to create a nice looking web UI, so I've never created a pull request. I'll happily contribute the code to nextflow-io/hack17#4 if that helps.
Phil Ewels
@ewels
Sep 06 2017 15:39
ok, nice! To be honest I was more thinking of generating a static HTML file to look at after completion of the pipeline in the issue, but having a running server to monitor pipelines in real time could be really nice too..
The nice looking web UI is the fun bit ;)
Mike Smoot
@mes5k
Sep 06 2017 15:46
My original idea was a dynamic version of the -with-dagoutput where the nodes change shape and color as they do work and complete. The dag would have to grow dynamically as nextflow computes the structure of the dag, so who knows how it would look. In any case, a standardized stream of events should allow lots of interpretations of the data. I am MORE than happy to delegate any UI design! :)
Phil Ewels
@ewels
Sep 06 2017 15:48
Ah interesting, someone in my group worked with something similar ages ago (not nextflow though). I think that there are some Javascript libraries kicking around which can render dag flow chart files in the browser, so it should be pretty doable I think..
Félix C. Morency
@fmorency
Sep 06 2017 17:15
Interesting project
We would probably be interested to help out
Paolo Di Tommaso
@pditommaso
Sep 06 2017 17:25
this is something that at some point we will implement
Félix C. Morency
@fmorency
Sep 06 2017 17:27
well, we can help ;)
Paolo Di Tommaso
@pditommaso
Sep 06 2017 17:27
thanks! I will keep you updated
Félix C. Morency
@fmorency
Sep 06 2017 17:54
and we're still waiting for that donation/beer money button on the NF web page
Paolo Di Tommaso
@pditommaso
Sep 06 2017 17:55
:)
Félix C. Morency
@fmorency
Sep 06 2017 19:20
one of my colleague placed a NF sticker in his car.
Paolo Di Tommaso
@pditommaso
Sep 06 2017 19:20
:joy:
Félix C. Morency
@fmorency
Sep 06 2017 19:21
ill try and grab a picture
Paolo Di Tommaso
@pditommaso
Sep 06 2017 19:21
let's do !
Simone Baffelli
@baffelli
Sep 06 2017 22:39
Where can I get nf stickers? I could place one on my bike 😂
Paolo Di Tommaso
@pditommaso
Sep 06 2017 22:40
if you came to NF workshop next week, you will have :)