These are chat archives for atomix/atomix

30th
Mar 2017
Stephen Carman
@hntd187
Mar 30 2017 01:49
@kuujo I wrapped everything in my state machine in try catches and I can't get the exception to bubble up or tell me any details about the issue at all
Jordan Halterman
@kuujo
Mar 30 2017 01:51
it’s probably happening inside the main Atomix state machine...
Stephen Carman
@hntd187
Mar 30 2017 01:56
any suggestion how I can track that down?
Like does my implementation look correct at a glance?
I'm almost wanna say it's Scala's fault, but that odesn't make any sense
Jordan Halterman
@kuujo
Mar 30 2017 02:00

@himanshug sorry your last question got lost to the cavern that is the top of my monitor. What you said is right. For safety, configurations are stored on disk. That complicates managing clusters that evolve a lot. The weird thing is, a node can crash and the cluster can evolve while it's down, but as long as at least one still existing and live node is in its configuration it will eventually find the new nodes and get the new configuration.

@jhalterman and I were actually just talking about how to replace a node's configuration but keep the logs. Currently, that's not possible but I think it can be made possible with relatively little work.

Anyways, you can remove a member of the cluster with Member#remove(), e.g. server.cluster().member(someAddress).remove(). TBH this isn't well tested. We haven't really done any work that uses that method, and looking at it I can't say it works. It should be fairly trivial to implement, in just not so sure that code is doing what it says it does right now :-P

Himanshu
@himanshug
Mar 30 2017 02:25
@kuujo even with Member#remove() , it means when a node permanently goes down ... cluster operator needs to send a "remove<crashed_node>" request to one of the live nodes which triggers Member#remove() ... that is not so operator friendly . One option is to have separate storage level for all 3 things (e.g. "the Log", "<term,vote> metadata" and "cluster configuration") .... then user can explicitly choose to store "cluster configuration" only in memory.... then removing a node permanently would mean , "bring down all the nodes and restart them all" , that would be same model as in zookeeper.... does that sound valid and would work in production? I can make that change.
Jordan Halterman
@kuujo
Mar 30 2017 02:43
That’s what’s done in ZooKeeper? IIRC ZooKeeper has a configurartion file that lists the nodes in the cluster. That’s not the case any more?
Himanshu
@himanshug
Mar 30 2017 02:44
@kuujo yeah, in case of ZK ...you would have to update configuration and restart them
that would be true in copycat case too
Jordan Halterman
@kuujo
Mar 30 2017 02:44
hmm
Himanshu
@himanshug
Mar 30 2017 02:44
on each node you would have "list of bootstrap addresses" in the config
if one of the bootstrap node goes permanently down
then one would need to update config on all the hosts before restarting
difference between ZK and copycat is that ZK needs "all servers" vs copycat needs "bootstrap servers" in the configuration
Jordan Halterman
@kuujo
Mar 30 2017 02:45
yeah that sounds right… let me think about it
gotta go put my son to sleep :-)
Himanshu
@himanshug
Mar 30 2017 02:46
np, we'll continue the discussion in "async" mode ... I gotta go sleep as well :)
Jordan Halterman
@kuujo
Mar 30 2017 03:10
@himanshug want me to sing you a song too? :-P
Jordan Halterman
@kuujo
Mar 30 2017 03:17
Alright, so the problem with doing this is configuration changes are necessarily managed through the log. Storing them in a separate file in Copycat is really just an optimization that allows it to read the last known configuration without scanning the entire log at startup. It's not really possible to separate the configuration from the log and still safely do configuration changes, so if the log is on disk then the configuration is ultimately on disk too even if the configuration file isn't.
aamargajbhiye
@aamargajbhiye
Mar 30 2017 04:43
I want to create distributed connection pool using Atomix. For connection, I want to create custom state machine with states such as Offline, Connecting, connected, Busy.
How can I achieve this using Atomix ?
Jordan Halterman
@kuujo
Mar 30 2017 05:03
That depends on what you expect the semantics of a distributed connection pool to be. I’m not sure I understand since connections are inherently tied to individual nodes. But Atomix is filled with state machine examples, and that’s what they’re there for. The simplest is DistributedValue. Define a Resource interface, define a ResourceStateMachine, define a set of commands, and implement methods on the state machine.
aamargajbhiye
@aamargajbhiye
Mar 30 2017 05:31
Thanks @kuujo , I will go through available examples. My distributed connection pool would not have connection objects. I intend to keep information about connections created on every node and their states. Like how many connections are created on particular node and what are their states etc. So that scheduler can use this information to schedule a task where connection is available.
Jordan Halterman
@kuujo
Mar 30 2017 05:32
I think we’re interested in improving the primitives that are provided by Atomix if there’s a data structure that can be used for this use case instead
this is a perfect use case for just a generic basic state machine API, but Atomix’s current API doesn’t really allow for something that rich to be defined
meaning, defining states and transitions
state machines are a lot more complex abstraction, but it would be cool to have a state machine resource that defines states and provides an API for submitting events/causing transitions
I guess that’s the primitive that’s needed
aamargajbhiye
@aamargajbhiye
Mar 30 2017 05:43
@kuujo It would be really nice to have a state machine resource, where different state machines could be created using state machine primitive just by specifying states and allowed transition. Helix provides this functionality.
But for now, I guess I will have to define custom state machine for connection using Resource and ResourceStateMachine interface
Himanshu
@himanshug
Mar 30 2017 14:07
@kuujo I see why configuration and log has to be at same storage level ... in my case of membership and leader election, I can probably afford to have "log + configuration" be in memory while only <term,vote> metadata be on disk (I can make that change to copycat to support this). However , would you recommend instead that I should not do that but just store everything on disk as that is more production tested path ?
@kuujo with DISK level, we can mimic ZK by removing the configuration protocol (or making it optional) from copycat and each node is given list of all the servers like in ZK.... then copycat would never know about configuration aside what is told on startup...... this wouldn't support "adding" more nodes without restarting all though.
Himanshu
@himanshug
Mar 30 2017 14:19
I'm probably gonna look a bit into elastic search which has custom implementation for membership and leader election... AFAIK, its not standard raft or paxos but a variant of gossip + bully algorithm + something.... and see how they manage cluster configuration. It probably does not provide as strong guarantees as raft does.... it is described at https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html , "Cluster State Updates" section looks very similar to raft where update is first forwarded to n number of other nodes and only then committed an applied.
btw, you said server.cluster().member(someAddress).remove() isn't tested ... how do you manage permanent node removal in your production setup ?
Jordan Halterman
@kuujo
Mar 30 2017 21:39
Lemme see if I can find it on my phone...
Jordan Halterman
@kuujo
Mar 30 2017 21:54
Yeah it's not used anywhere. This is actually something I have on my todo list for our project right now. It doesn't handle an evolving cluster very well, and I have to look in to some issues with that. I didn't write the cluster manager and haven't had time to look into it yet, but you can feel free to poke around if you want :-)