These are chat archives for atomix/atomix

1st
May 2016
Roman Pearah
@neverfox
May 01 2016 00:23

@davidmoravek Running into a bit of a problem with running a cluster inside of Docker with a client living outside of Docker. I have exposed and published ports for each replica and I'm having the client initiate the connection there, and the initial connection goes fine, but then I believe the replica's are advertising their location to the client as the container host:port (which of course, isn't accessible from the outside) and that causes the client to immediately start failing on the first KeepAliveRequest:

20:10:05.689 [copycat-client-io-1] DEBUG i.a.c.client.session.ClientSession - Registering session: attempt 1
20:10:05.690 [copycat-client-io-1] DEBUG i.a.c.client.session.ClientSession - Sending RegisterRequest[client=a25cc223-1b28-49f6-841a-d15d4b197f02]
20:10:05.691 [copycat-client-io-1] DEBUG i.a.c.client.util.ClientConnection - Connecting to /192.168.99.100:32788
20:10:05.714 [copycat-client-io-1] DEBUG i.n.util.internal.ThreadLocalRandom - -Dio.netty.initialSeedUniquifier: 0xa05547530a656cca (took 0 ms)
20:10:05.748 [copycat-client-io-1] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: unpooled
20:10:05.748 [copycat-client-io-1] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 65536
20:10:05.748 [copycat-client-io-1] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.maxThreadLocalCharBufferSize: 16384
20:10:05.779 [copycat-client-io-1] DEBUG i.a.c.client.util.ClientConnection - Setting up connection to /192.168.99.100:32788
20:10:05.782 [copycat-client-io-1] DEBUG i.a.c.client.util.ClientConnection - Sending ConnectRequest[client=a25cc223-1b28-49f6-841a-d15d4b197f02]
20:10:05.785 [copycat-client-io-1] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacity.default: 262144
20:10:05.794 [copycat-client-io-1] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
20:10:05.797 [copycat-client-io-1] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
20:10:05.797 [copycat-client-io-1] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
20:10:06.069 [copycat-client-io-1] DEBUG i.a.c.client.util.ClientConnection - Received ConnectResponse[status=OK, leader=7061bafaa593:5000, members=[7061bafaa593:5000]]
20:10:06.090 [copycat-client-io-1] DEBUG i.a.c.client.session.ClientSession - Received RegisterResponse[status=OK, session=79, leader=7061bafaa593:5000, members=[7061bafaa593:5000]]
20:10:06.092 [copycat-client-io-1] DEBUG i.a.c.client.DefaultCopycatClient - State changed: CONNECTED
20:10:06.093 [copycat-client-io-1] INFO  i.a.c.client.session.ClientSession - Registered session 79
20:10:06.094 [copycat-client-io-1] DEBUG i.a.c.client.session.ClientSession - 79 - Sending KeepAliveRequest[session=79, commandSequence=0, eventIndex=79]
20:10:06.099 [copycat-client-io-1] DEBUG i.a.c.client.util.ClientConnection - Connecting to 7061bafaa593:5000
20:10:06.103 [copycat-client-io-1] DEBUG i.a.c.client.session.ClientSession - 79 - Sending CommandRequest[session=79, sequence=1, command=io.atomix.manager.internal.GetResource@7581fa62]
20:10:06.105 [copycat-client-io-1] DEBUG i.a.c.client.util.ClientConnection - Failed to connect to the cluster
20:10:06.105 [copycat-client-io-1] DEBUG i.a.c.client.session.ClientSession - 79 - Sending KeepAliveRequest[session=79, commandSequence=0, eventIndex=79]
20:10:06.106 [copycat-client-io-1] DEBUG i.a.c.client.util.ClientConnection - Connection closed
20:10:06.106 [copycat-client-io-1] DEBUG i.a.c.client.util.ClientConnection - Connecting to 7061bafaa593:5000
20:10:06.107 [copycat-client-io-1] DEBUG i.a.c.client.util.ClientConnection - Failed to connect to the cluster
20:10:06.108 [copycat-client-io-1] DEBUG i.a.c.client.DefaultCopycatClient - State changed: SUSPENDED

I did notice above that you did something with your setup to get the cluster to report something like docker.loc/172.17.0.2:1234. How did you do that and what exactly is it doing?

It's possible that there's no way around this given that the want clients maintain connections is by using the information returned by the cluster, which has no way I can think of of being Docker-aware about the networking that's setup with the outside world. Or maybe there is but my Docker-fu is sorely lacking.
Jordan Halterman
@kuujo
May 01 2016 01:50
interesting
yeah as much as I have enjoyed my prior experiences with Docker I’m sure my Docker-fu is lacking too
but I’m really interested in figuring this out
Roman Pearah
@neverfox
May 01 2016 01:52
Everything works fine if the client is also inside docker, so that everyone is on the same network
but across a bridge, no go
Jordan Halterman
@kuujo
May 01 2016 01:52
gotcha
Roman Pearah
@neverfox
May 01 2016 01:53

I believe the problem is basically because of this:

Once the client connects to the cluster and opens a session, the client will receive an updated list of servers to which to connect.

Jordan Halterman
@kuujo
May 01 2016 01:53
right… and those addresses only make sense from the perspective of the servers inside the container?
Roman Pearah
@neverfox
May 01 2016 01:53
right
Jordan Halterman
@kuujo
May 01 2016 01:53
damn
Roman Pearah
@neverfox
May 01 2016 01:54
Connecting to /192.168.99.100:32788
that's the initial connection
But then later Connecting to 7061bafaa593:5000
that's the container hostname and port
Jordan Halterman
@kuujo
May 01 2016 01:55
ahh yeah
thinking
A couple of things to note… Copycat servers have support for exposing separate client/server Addresses. If separate client/server addresses/servers are used, servers will communicate with one another through the server Addresses and clients will only ever see the client Addresses. However, AtomixReplicas use the client addresses for working with resources.
Secondly, CopycatClient can be configured with a ServerSelectionStrategy that can be used in a hacky way to override the Addresses provided by the cluster.
I’m not sure if either of those pieces of information are useful, and both certainly seem like approaches to fix symptoms of the problem rather than the problem.
Roman Pearah
@neverfox
May 01 2016 02:00
I can see how that would be useful yes
Jordan Halterman
@kuujo
May 01 2016 02:00
Technically, you could override the Address sent back by the cluster with one the client can actually connect to, but it’s pretty hacky
Roman Pearah
@neverfox
May 01 2016 02:01
Compounding the problem is that you can't really make a Docker container aware of how you've bridged it
Jordan Halterman
@kuujo
May 01 2016 02:01
right
yeah that sucks
Roman Pearah
@neverfox
May 01 2016 02:04
I think you'd have to set the hosts file on the client's network
or something like that
Jordan Halterman
@kuujo
May 01 2016 02:14
I think Docker’s definitely on my short list of things to fight with if it’s not figured out before I can get to it. I’m really interested if any changes need to be made in Copycat/Atomix. In the meantime, I’m about to submit a bunch of PRs for the bug fixes... So at least that’s something :-)
Roman Pearah
@neverfox
May 01 2016 02:15
:)