These are chat archives for nextflow-io/nextflow

8th
Aug 2016
Mokok
@Mokok
Aug 08 2016 11:54

Hi @pditommaso , i was playing with nextflow again, and specially about mixing languages (in the "script" block). It seems that it's restrained to conditioning bash exec with groovy control flow (if/else/switch). As far as i can see, groovy is executed locally and then the shell is ran by the executor.

1- Is there any way to code more than conditioning (modification/preparation of produced data to push into output channel?
2- Additionally, how to execute groovy code on the executor side ?

3- Bug ? If I put some groovy code after a shell block, groovy is executed, but the shell is evaluated ass 'unknown command null' : NextFlow doesn't report it as a script syntax error. Is it a bug or am i doing it wrong ?

Paolo Di Tommaso
@pditommaso
Aug 08 2016 12:33
2) native process i.e. groovy remote processes are supported only by the Ignite executor
1 & 3 not clear
Mokok
@Mokok
Aug 08 2016 12:42

a script block such as :

script:
println "something"
"""
aShellCmd
"""
println "somethingelse"

what i mean is i can't do this, the 2 groovy lines are well executed but not the shell
after some tries, it appears that i can't put groovy after a shell call
Paolo Di Tommaso
@pditommaso
Aug 08 2016 12:45
in the script you can use groovy code, but it's meant only to compose the script command (string) in a dynamic manner
i.e. is evaluated always locally, only the command string is executed remotely
it appears that i can't put groovy after a shell call
you can, but it's useless
Mokok
@Mokok
Aug 08 2016 12:47
yep, saw that on the doc (about control-flow 'only' use of groovy if mixing languages), i was just wondering if it was planned to be changing
Paolo Di Tommaso
@pditommaso
Aug 08 2016 12:48
can you provide me the link . .
Mokok
@Mokok
Aug 08 2016 12:49
in case that the shell script produces such results that you have to remodel/parse/modify it before sending it to the output channel
Paolo Di Tommaso
@pditommaso
Aug 08 2016 12:53
I see. Nextflow is a DSL built on top of the Groovy programming lang, thus control-flow and other statement are specified in Groovy, not other lang is planned
Mokok
@Mokok
Aug 08 2016 12:54

mh ok !

then the 3- is pointless ^^

Paolo Di Tommaso
@pditommaso
Aug 08 2016 12:55
:)
Mokok
@Mokok
Aug 08 2016 12:55
thk for the answer as always
Paolo Di Tommaso
@pditommaso
Aug 08 2016 12:56
were you hoping to use a different programming lang for that?
Mokok
@Mokok
Aug 08 2016 12:59
not really, it was more about my own curiosity, and about doing pre/post-process (like a groovy sandwich with shell inside), but as you said, that's not intended to work like that ^^
Paolo Di Tommaso
@pditommaso
Aug 08 2016 13:00
:+1:
Mike Smoot
@mes5k
Aug 08 2016 17:10
@pditommaso Just following up on your comment here nextflow-io/nextflow#211 I just branch from master and assuming I get things working make a pull request like normal, correct? Or is there a different protocol you'd prefer?
Paolo Di Tommaso
@pditommaso
Aug 08 2016 17:16
@mes5k yes, it would be better if you commit your changes on a separate branch, thus the PR will be created that branch on the nextflow origin as well
Paolo Di Tommaso
@pditommaso
Aug 08 2016 17:22
I'm looking for a strategy to grow/shrink a pool of VM instances dynamically, depending the pipeline workload.
Does anyone have any suggestion to implement it?
Mike Smoot
@mes5k
Aug 08 2016 17:37
Sorry to bother, but haven't I done that in the past? I'm hardly a git or github guru, so if I've gotten it wrong in the past I'd like to figure it out.
Paolo Di Tommaso
@pditommaso
Aug 08 2016 17:38
Not a big problem, but I think you didn't. Let me check
nextflow-io/nextflow#204
pditommaso merged 2 commits into nextflow-io:master from mes5k:splitFasta_single 17 days ago
um, it looks that you worked on mes5k:splitFasta_single
so you did :)
Mike Smoot
@mes5k
Aug 08 2016 17:40
Ok good. Otherwise I would have been very confused! :)
I'll follow the same procedure on this and subsequent tickets.
Paolo Di Tommaso
@pditommaso
Aug 08 2016 17:41
but I don't think to have the option to choose to what branch to merge
Mike Smoot
@mes5k
Aug 08 2016 17:41
Because I branched from master?
Or because the pull request was for master?
Paolo Di Tommaso
@pditommaso
Aug 08 2016 17:42
I don't think so, maybe it is possible to choose it when the PR is created?
let me check
Paolo Di Tommaso
@pditommaso
Aug 08 2016 17:50
ok, when you create the PR you can choose to which branch to merge it
but you cannot create a new branch on the origin
thus I will create a new one for that
@mes5k Ok, I've create a fix_issue_211 branch
Mike Smoot
@mes5k
Aug 08 2016 17:54
Sounds good, I'll make the PR for fix_issue_211!
Paolo Di Tommaso
@pditommaso
Aug 08 2016 17:54
great! thx!
Mike Smoot
@mes5k
Aug 08 2016 19:42
Hi Paolo, do you have an estimate on when release 0.22.0 might come out? No rush, just curious?
Paolo Di Tommaso
@pditommaso
Aug 08 2016 19:44
Let's say before the end of the month
Mike Smoot
@mes5k
Aug 08 2016 19:44
Sounds great - I have some very excited users!
Paolo Di Tommaso
@pditommaso
Aug 08 2016 19:45
since it includes some important changes, I need to be sure regarding some little details
ah good
are they your workmates?
Mike Smoot
@mes5k
Aug 08 2016 19:49
Yes, the computational biologists in my group. Once I get the pipelines "working" they refine and maintain them. We're interested in the clean command so that we can both archive as little as possible, but also maintain enough state in a pipeline such that we'll be able to re-run a given pipeline with partial new data and (hopefully) only re-run what is necessary.
Paolo Di Tommaso
@pditommaso
Aug 08 2016 19:50
ok, so you need a production stable version, right?
Mike Smoot
@mes5k
Aug 08 2016 19:58
Yeah, I mentioned to this user that he could try the snapshot, but he wasn't interested. He's fine waiting for something stable.
Paolo Di Tommaso
@pditommaso
Aug 08 2016 19:58
OK
BTW what cluster are u using?
Mike Smoot
@mes5k
Aug 08 2016 20:02
At the moment everything is running locally on dedicated machines. Once we're ready to scale out I expect that we'll begin by using the default Ignite.
Paolo Di Tommaso
@pditommaso
Aug 08 2016 20:03
I see, any interested on AWS cloud deployment ?
Mike Smoot
@mes5k
Aug 08 2016 20:04
Yes, that's most likely where we'd scale out to.
Paolo Di Tommaso
@pditommaso
Aug 08 2016 20:05
ok, interesting .. working hard on that ;)
Mike Smoot
@mes5k
Aug 08 2016 20:07
I thought Ignite already worked on AWS? Just set up an S3 bucket in lieu of multicast? Easy right? :)
Paolo Di Tommaso
@pditommaso
Aug 08 2016 20:08
it works, but I'm testing the new EFS shared file system + improving process resources allocation
Mike Smoot
@mes5k
Aug 08 2016 20:10
Sounds good - no immediate rush from us