These are chat archives for nextflow-io/nextflow

20th
Jan 2015
Paolo Di Tommaso
@pditommaso
Jan 20 2015 00:35
What GridGain or ClusterK?
Andrew Stewart
@andrewcstewart
Jan 20 2015 00:40
GG
(but both)
Paolo Di Tommaso
@pditommaso
Jan 20 2015 00:41
Oh, GridGain is a very cool technology
It's an in memory clustering and streaming technology
think more similar to Apache Sparck than Star cluster/SGE
I'm using in nextflow to submit jobs to remote nodes
however there's still a lot to do
do not aspect that it stable
ClusterK instead is much more similar to SGE
but it take advantage of cloud features, like spot instances, elastic load balancing, etc
Andrew Stewart
@andrewcstewart
Jan 20 2015 00:46
ah
why GG and not just Spark ?
Paolo Di Tommaso
@pditommaso
Jan 20 2015 00:46
it looks promising and mor easy to integrate with Nextflow
well, that's a good question
Andrew Stewart
@andrewcstewart
Jan 20 2015 00:48
both GG and ClusterK look like they are proprietary as well?
Paolo Di Tommaso
@pditommaso
Jan 20 2015 00:48
however I think GG is superior as clustering solution
Spark became very popular because they invested a lot of money on it..
GG is open source, and it is going to become an Apache project,
s
as Apache Ignite
ClusterK is a commercial platform
Have a look to this
Andrew Stewart
@andrewcstewart
Jan 20 2015 00:54
uggggh.. you're making me reevaluate everything I know about cluster computing :D
Paolo Di Tommaso
@pditommaso
Jan 20 2015 00:55
:)
I haven't evaluated yet, but the features they claim look very interesting
Andrew Stewart
@andrewcstewart
Jan 20 2015 01:45
do you see StarCluster becoming an executor itself?
Paolo Di Tommaso
@pditommaso
Jan 20 2015 01:47
I would say no
I'm unhappy how Starcluster scale
It needs a lot of time to deploy instances
and above all the NFS file system it deploys is useless with > 10 nodes
Andrew Stewart
@andrewcstewart
Jan 20 2015 01:50
the nfs deployment is def the weak spot
cirrus is interesting though
Paolo Di Tommaso
@pditommaso
Jan 20 2015 01:58
I will keep you update on this
I'm going off line
bye
Andrew Stewart
@andrewcstewart
Jan 20 2015 18:14
thanks!
Andrew Stewart
@andrewcstewart
Jan 20 2015 21:50
Hey, have any of you done any slidedecks about NextFlow? Im curious to see if anyone has any graphical summaries of what it is for a non-technical audience.
Paolo Di Tommaso
@pditommaso
Jan 20 2015 22:26
Recently a gave a presentation for non-technical people
Have a look to this link
let me know if it fits your need
@andr
Also you may be interested in this poster
Andrew Stewart
@andrewcstewart
Jan 20 2015 22:32
awesome, thanks!
Paolo Di Tommaso
@pditommaso
Jan 20 2015 22:35
Obrigado :)