These are chat archives for nextflow-io/nextflow

1st
May 2018
Stephen Kelly
@stevekm
May 01 2018 16:19
hey I am trying to envision how to use Nextflow as part of something like a web dashboard to set up and run analyses, has anyone done this? I want to be able to like view my datasets on the web app GUI, choose which ones to process, which pipeline & configs to use, and then start the NF pipeline from the GUI and get stats on the task execution status. I was thinking some kind of API might be useful? Were there any considerations for features like that? I could do a wrapper script around the stderr & stdout and trace file, and execute by just running the system CLI commands, but that does not seem very robust.
Also was not sure if this was something that nextflow run could potentially be able to do, or if you would need another process that launches and monitors nextflow run
Alexander Peltzer
@apeltzer
May 01 2018 16:21
My colleague Sven F is working on something in that direction
@sven1103 on github if you want to get in touch
Stephen Kelly
@stevekm
May 01 2018 16:33
thanks!
working on it right now
not an API, but trace info and wf info on start/end/fail will be send via HTTP POST with a -with-messages option
Stephen Kelly
@stevekm
May 01 2018 16:37
very cool
how do you receive the messages?
Sven F.
@sven1103
May 01 2018 16:38
web service
atm
but you are free to implement on the receiving side what you want
as long as you parse the JSON object
:)
at QBiC we will dump it in a DB and then do some monitoring for users
Stephen Kelly
@stevekm
May 01 2018 16:44

yeah thats like exactly what I am trying to do

at QBiC we will dump it in a DB and then do some monitoring for users

havent done a lot of web stuff, but I am guess you can use like node.js to listen & dump the messages to db then have the client app get the status information from the db?
Sven F.
@sven1103
May 01 2018 16:46
exactly
I want wo keep it very simple
Stephen Kelly
@stevekm
May 01 2018 16:47
do you initiate the Nextflow pipeline by system calls to nextflow run? Is that even possible from client JS app?
I guess it would have to be done from the server side app
Sven F.
@sven1103
May 01 2018 16:47
ah for this I dont have a good solution yet
Stephen Kelly
@stevekm
May 01 2018 16:48
Ok
maybe an Rshiny or Python Dash would be an easy prototype
plus backend something
Sven F.
@sven1103
May 01 2018 16:48
we are using a portal environment and would need to allow submission from a web portlet
but this is the long term task for 2018
Stephen Kelly
@stevekm
May 01 2018 16:53
Is this http messaging feature mature enough to be integrated into the main Nextflow program?
Sven F.
@sven1103
May 01 2018 16:54
not yet
but I am going to create a PR as soon as I think it is
will need feedback from more experienced programmers then :D
John Hadish
@JohnHadish
May 01 2018 17:31

Hello everyone,
I had a quick question that I feel has probably already been answered somewhere, but which I cannot find. I have a process that takes a file input from another process using a glob pattern. This process does not use this file, but it needs to go into the outpput channel (using the same glob pattern). According to the NF documentation:

"Some caveats on glob pattern behavior:

Input files are not included in the list of possible matches."

I was wondering if there is an easy way around this without having to make a new channel
John Hadish
@JohnHadish
May 01 2018 17:36
(It should also be noted that this particular process does not produce any files of its own, only a value. Also, I am already using my standard out in my script)
process SRR_to_SRX {
  module 'python3'
  tag { sra }
  input:
    set val(sra), file("${sra}_?.fastq") from srx_value
  output:
    set stdout, sra, file("${sra}_?.fastq") into raw_fastq
  """
  python3 ${PWD}/scripts/retrieve_sample_metadata.py $sra
  """
}
Stephen Kelly
@stevekm
May 01 2018 17:39
something like:
input:
file(input_files: '*') from input_channel

output:
file(input_files)
I think that would work??
input:
file(input_files: "${sra}_?.fastq") from input_channel

output:
file(input_files)
maybe try that I think?
John Hadish
@JohnHadish
May 01 2018 17:56
process SRR_to_SRX {
  module 'python3'
  tag { sra }
  input:
    set val(sra), file(pass_files) from srx_value
  output:
    set stdout, sra, file(pass_files) into raw_fastq
  """
  python3 ${PWD}/scripts/retrieve_sample_metadata.py $sra
  """
}
Thanks a bunch Stephan, your response made me realize I do not actually need to use the glob pattern, since no files are created in this process. I can just define it like above.
Stephen Kelly
@stevekm
May 01 2018 19:02
@sven1103 how do I go about installing this version of NF from your github then?
Sven F.
@sven1103
May 01 2018 19:19
@stevekm you clone, checkout the branch nfbroadcast, then in the root dir you do makeand then execute launch.sh
you need to have Gradle installed
gawells
@gawells
May 01 2018 22:20
Is there a way to use a regex as a dynamic criteria for collectFile? I want to group outputs based on the presence of a substring
Mike Smoot
@mes5k
May 01 2018 22:28
@gawells looking at this I think you should be able to write a closure that uses a regex on item and then generates the proper output tuple. Maybe write a separate function to capture the logic and then call it from collectFile.