These are chat archives for atomix/atomix

11th
Apr 2018
Daniel Tam
@dualscyther
Apr 11 2018 00:07
I was wondering if the Catalyst project is still good to use or whether atomix2 has better functionality? I just need it for high level communication where I can give it ports and ips in the constructor and it will automatically handle most things.. from what I can tell it looks like NettyTransport does the job?
Jordan Halterman
@kuujo
Apr 11 2018 00:36
Yep, I think optional should work and the jaxrs dependency should be needed only if someone’s adding a primitive with a rest resource
Jordan Halterman
@kuujo
Apr 11 2018 00:42
@dualscyther I suppose that depends on what you mean by most things? Catalyst had a connection-based abstraction that was used in Copycat. Atomix 2 has more generic APIs for direct messaging, unicast/multicast/broadcast, publish-subscribe (decoupling producers from consumers), group membership, and storage/replication. All the messaging/cluster management features can certainly be used without storage/replication as well. Atomix 2.1 is really a much more complete framework for building distributed systems
Daniel Tam
@dualscyther
Apr 11 2018 01:15
ahh ok yeah, I'm looking for something that just does generic point to point communication
being able to have some broadcast abstraction over the point to point connections would be a bonus
would be preferable if it could be used with no awareness of its use/being in a cluster (as in, literally just a messaging library)
You should check out Vert.x, which is a project I used to work on that inspired some of the APIs like the above
Daniel Tam
@dualscyther
Apr 11 2018 01:31
thanks!
Jordan Halterman
@kuujo
Apr 11 2018 01:37
The equivalent in Atomix would be:
Atomix atomix = Atomix.builder()
  .withMulticastEnabled()
  .build();

atomix.start().join();

atomix.eventingService().subscribe(“test”, message -> {
  return “Hello world back!”;
});

atomix.eventingService().send(“test”, “Hello world!”).thenAccept(response -> {
  ...
});
Jordan Halterman
@kuujo
Apr 11 2018 03:44
@johnou that’s the last big PR. Just working on the tests from here on.
Next week I’ll start the ONOS integration which should accelerate the process a lot.
Need to get the test coverage up to 70% and passing consistently to release it
Jordan Halterman
@kuujo
Apr 11 2018 03:49
I’ll also start rewriting the website along with that work
Daniel Tam
@dualscyther
Apr 11 2018 04:19
thanks for the vertx recommendation, looks like it's probably the thing I need!
Jordan Halterman
@kuujo
Apr 11 2018 04:19
yep it’s a really cool project
good community too
good luck!
Johno Crawford
@johnou
Apr 11 2018 06:46
@kuujo and the netty timeout fixes from ONOS right?
Did you see my question above btw?
Jordan Halterman
@kuujo
Apr 11 2018 13:12
Oh yeah I saw it and never answered it :-) yes
Nobin Mathew
@nmathew
Apr 11 2018 13:12
I am building the code in this gist against 2.1.0-SNAPSHOT, but failing with some erros. Errors are pasted in same gist. Any help
I am building on windows 7 machine with java 8
Jordan Halterman
@kuujo
Apr 11 2018 13:17
Primary-backup essentially uses a leader election to elect a primary and backups for each partition. Primitive changes are sent to the primary for the partition and replicated to backups. If a primary crashes then the next backup will take over and replicate the state to an extra node to replace the original primary.
One sec
Johno Crawford
@johnou
Apr 11 2018 14:06
Ok so using MultiPrimaryProtocol will give me primary backup I guess
Abd
Ffs mobile
Johno Crawford
@johnou
Apr 11 2018 14:24
And a map has one partition unless that sharded one is used, right?
PartitionedAsyncConsistentMap
Johno Crawford
@johnou
Apr 11 2018 14:33
@nmathew try swapping them to core nodes
Nobin Mathew
@nmathew
Apr 11 2018 14:38
ok
Johno Crawford
@johnou
Apr 11 2018 14:39
The bootstrap nodes that is
The others can stay data
Nobin Mathew
@nmathew
Apr 11 2018 14:40
ok
cool, thanks
Johno Crawford
@johnou
Apr 11 2018 14:42
Does that work or does it still fail?
Nobin Mathew
@nmathew
Apr 11 2018 14:42
works
Johno Crawford
@johnou
Apr 11 2018 14:42
:+1:
Johno Crawford
@johnou
Apr 11 2018 14:50
@kuujo shouldn't it work with just data nodes though
Nobin Mathew
@nmathew
Apr 11 2018 15:30
this is removed io.atomix.messaging.Endpoint?
with latest master
2.1.0 beta2 seems to have it, I will use that
Nobin Mathew
@nmathew
Apr 11 2018 15:43
with 2.10 beta CORE is not working, DATA works
beta2
Johno Crawford
@johnou
Apr 11 2018 15:46
@nmathew what about with local snapshot of master
Nobin Mathew
@nmathew
Apr 11 2018 15:59
what happended is I synced again
So I lost that snapshot
with latest Endpoint is gone
Johno Crawford
@johnou
Apr 11 2018 16:32
yeah replaced with Address iirc
Jordan Halterman
@kuujo
Apr 11 2018 17:38
Yes it should work with data nodes. I think this may be one of the bugs I fixed in that big PR
Also fixing the rest of the test issues...
Buncha bootstrap bugs that will be fixed in that PR
Jordan Halterman
@kuujo
Apr 11 2018 17:44
Also, all primitives are always partitioned if they can be (e.g. maps, sets, anything else that has a hashable key. If one wanted to create a single partition primitive then they’d have to store it in a partition group with one partition
Johno Crawford
@johnou
Apr 11 2018 17:55
Partition size calculated depending on the amount of data / core nodes or maybe another tunable config
?
Johno Crawford
@johnou
Apr 11 2018 18:38
@kuujo I also dropped some review comments on that pr
Jordan Halterman
@kuujo
Apr 11 2018 20:02

It wouldn’t be wise to set the number of partitions based on data size because repartitioning data can be very expensive. So, the number of partitions is fixed and defined by the user, but there’s not much overhead to a data partition, so lots of data partitions allows for the cluster to scale to large sizes without repartitioning.

Repartitioning would also be very difficult using the state machine model where the system itself has no concept of how data is partitioned. It would also be difficult to preserve strong consistency when repartitioning, so it’s just not supported and dunno if it ever will be.

Just need large defaults

Just working through one more bug in the cluster manager to at least get it back to where it was
Johno Crawford
@johnou
Apr 11 2018 20:58
@kuujo did you say there were timeout fixes from ONOS too?