These are chat archives for nextflow-io/nextflow

29th
Jul 2016
Sandeep Shantharam
@machbio
Jul 29 2016 01:31
#208 raised for the parameter through json/yaml file
Mokok
@Mokok
Jul 29 2016 09:59

is there a way to run multiple .nf file at once ?

i mean i can do something like that but it runs NextFlow several times:
find /Dir/To/Execute/All/Contained/NF/Files -maxdepth 1 -type f -exec ./nextflow run {} [options] \;

i find it ugly ^^

Paolo Di Tommaso
@pditommaso
Jul 29 2016 10:00
you can write a script that launches other NF scripts
Mokok
@Mokok
Jul 29 2016 10:01
by calling "nextFlow run script" in a bash command inside an nf script ?
Paolo Di Tommaso
@pditommaso
Jul 29 2016 10:05
yes
Mokok
@Mokok
Jul 29 2016 10:09
but isn't it the same ? it'll launch several instancies of nextFlow, one for each script huh?
the perfect behavior would be to put the scripts one after the other as if it was a single script with unrelated taskflow
Paolo Di Tommaso
@pditommaso
Jul 29 2016 10:11
not possible at this time
Mokok
@Mokok
Jul 29 2016 10:11
ok, still good to know ;)
Paolo Di Tommaso
@pditommaso
Jul 29 2016 10:12
if they are unrelated scripts, you can try to merge in a single one and launch in a single run
cat *.nf | nextflow run -
Mokok
@Mokok
Jul 29 2016 10:12
good idea ! thk
Paolo Di Tommaso
@pditommaso
Jul 29 2016 10:13
but potentially dangerous is your have overlapping identifiers, params, process names, etc ..
Mokok
@Mokok
Jul 29 2016 10:14
yep, it could, but then i'd jsut need to parse and check. i think it's still a better approach than a multiple nf run
Paolo Di Tommaso
@pditommaso
Jul 29 2016 10:15
:+1:
Mokok
@Mokok
Jul 29 2016 12:28
does Channel.watchPath support anything ? local, ftp, sftp ?
It's not mentioned, but i'm quite sure it does.
Paolo Di Tommaso
@pditommaso
Jul 29 2016 12:29
it supports local file system and AWS S3
a contribution for http/ftp is welcome !
;)
Mokok
@Mokok
Jul 29 2016 12:34
ok, i note that in case I (or who ever will read my note) implement this feature
to be shared with NextFlow little soldiers, they'd appreciate it much
Paolo Di Tommaso
@pditommaso
Jul 29 2016 12:36
wat?! :)
Mokok
@Mokok
Jul 29 2016 12:37
just a note to share the code with you if we implement the watchPath for ftp/http
Paolo Di Tommaso
@pditommaso
Jul 29 2016 12:37
ok
that would be useful, it would provide transparent access to http/ftp resources as a plain file system, not just for watchPath
Mokok
@Mokok
Jul 29 2016 14:44
i'd humbly notice you it seems that java.nio.file.WatchService can handle CREATE event when a file comes to the watched ftp path. I only did a really short research. I hope it can, somehow, help.
Paolo Di Tommaso
@pditommaso
Jul 29 2016 14:46
it's surely not a trivial thing to implement
but I've never tried, I have no details
Mokok
@Mokok
Jul 29 2016 14:49
It's surely not, indeed. It rarely is ^^
Paolo Di Tommaso
@pditommaso
Jul 29 2016 14:50
:)
Mike Smoot
@mes5k
Jul 29 2016 16:38

@pditommaso in addition to what nextflow-io/nextflow#208 does for params, I've been doing something similar between processes and channels: my process outputs a yaml file that I then parse into useful data with various maps and flatMaps. For my use case this is a pretty straightforward way for a process to return complicated data. While this isn't much of a burden, I've been pondering whether something like a parseYAML or parseJSON operator might be useful. Or possibly something like

output:
     yaml(stdout) into newChannel

There's no pressing need here, but I've found it a useful pattern and thought I'd share the idea.

Paolo Di Tommaso
@pditommaso
Jul 29 2016 17:32
@mes5k Sounds interesting, could you please share an example?
Mike Smoot
@mes5k
Jul 29 2016 17:35
Will do - I'm pressed for time today, but should be able to put something together next week.
Paolo Di Tommaso
@pditommaso
Jul 29 2016 17:35
great, no rush !