real-logic/Aeron

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

John Davis

@unicomp21

Has anyone taken a look at Seastar and co_await? For Aeron? The redpanda kafka replacement is benefiting greatly from it.

Michael Szymczak

@michaelszymczak

Hi, I have noticed some interesting and potentially dangerous use case in how Artio treats numeric constants such as ZERO. I did not know what to make of it, so I created an issue for that real-logic/artio#452

1 reply

Vachagan Balayan

@vach_b_twitter

Hi Everyone, i'm having some hard time with aeron cluster, are there any books resources that could take alook before buggin you here?

i'm reworking a core services for an exchange (every essential service + matching engine) to achieve insane speeds and TPS,
however this one is crypto exchange and they cant just restart things over weekend like banks, so i'm trying to dive into Raft via aeron
but there are some things i cant see how to control and configure...

Michael Barker

@mikeb01

Hi @vach_b_twitter, we don't do Aeron support on this channel (it is for community contribution discussions). You can contact Real Logic to arrange support and consulting (http://real-logic.co.uk). Or you have specific questions on Stack Overflow (using the aeron tag). There is also a tutorial available: https://github.com/real-logic/aeron/wiki/Cluster-Tutorial.

Vachagan Balayan

@vach_b_twitter

thanks, sent an email )

zaradai

@zaradai

I have a gripe with the generated c SBE code for the template id. As an example it generated
SBE_ONE_DEF uint16_t events_instrumentLoad_sbe_template_id(void)
{
return (uint16_t)16;
}

this isn't compile time constant so i am unable to a switch in a fragment handler. I would prefer a simple switch to handle different incoming events vs a hash lookup, am i missing something?

zaradai

@zaradai

just noticed comment above about SBE discussions, please ignore I'll go to SBE.

Niel de Wet

@nieldw

With no changes to our code, but just a restart, we are suddenly facing an unusual ERROR when the Consensus Module is starting up. What could be possible causes? Full stack trace in thread.

ERROR io.aeron.cluster.ConsensusModuleAgent java.lang.IllegalStateException: Cannot assign long arg5 to int

2 replies

Martin Thompson

@mjpt777

@MarshalOfficial This channel is for contribution discussions and not for free support.

Bill Segall

@billsegall

Hi @mjpt777

It's been a while but I'm now looking at adding MDC support to aeron-go.

I was wondering if there is an XML spec for the flyweights used for AddDestination etc? I had a bit of a browse in the aeron git and wiki but failed to find anything.

It all seems pretty transparent from the code - it's just I prefer to start from a spec where possible.

Thanks,
Bill.

Todd L. Montgomery

@tmontgomery

Hi Bill. No spec other than the Javadoc and code for the messages between the clients and driver.

Bill Segall

@billsegall

cool. Thanks!

Todd L. Montgomery

@tmontgomery

if you do have questions, let us know. It's fairly simple and (I hope) straight forward, but more than happy to confirm/deny anything we can.

Konstantin Dmitriev

@konstantind23_gitlab

Is using ReplayMerge necessary for merging replay with live stream? I guess otherwise it is not trivial to find the
cut off point at which subscriber should consume live stream instead of replayed and guarantee "at most once" delivery.

The problem I am facing is that ReplayMerge does not seem to work where live messages are published and recorded over multicast. I am getting error message "channel does not allow manual control" which I guess means ReplayMerge is tring to add subscription to the publisher which publishes over multicast channel that does not support manual control.

All ReplayMerge examples/tests use MDC so I wonder if multicast is supported for this use case

Todd L. Montgomery

@tmontgomery

@konstantind23_gitlab ReplayMerge supports multicast on the liveDestination. The subscription must be a manual one for MDC and multicast.

Konstantin Dmitriev

@konstantind23_gitlab

Thank you, I can get ReplayMerge working now for single publication sessionId. Unfortunately it does not seem to support multiple simultaneous sessions. I have multiple clients publishing to the same streamId over the same multicast channel, each having different sessionId. I tried running one ReplayMerge per session (getting sessionIds from archive recordings) with segregated replayDestinations but ReplayMerge gets confused when attempting to switch to live as it is trying to merge replay that delivers messages from a single session with live stream that delivers events from multiple sessions.

Ideally I would have one recording per streamId instead of the current recording per sessionId. Forcing the shared sessionId on multiple publishers results in archive/subscription treating multiple publications as one session, i.e. removing what it thinks are duplicate message.

Todd L. Montgomery

@tmontgomery

To do what you want, you would need to use session specific subscriptions.

@konstantind23_gitlab This channel is for contribution discussions and not for free support.

Bill Segall

@billsegall

Hi again, I'm a little confused and was hoping to get soem clarity regarding the ChannelEndpointStatus as I'm needing to implement this the golang ReplayMerge.

We seem to have 5 states defined, INITIALIZING = 0, ERRORED = -1, ACTIVE = 1, CLOSING = 2. and then curiously NO_ID_ALLOCATED which is also -1 but an int rather than a long.

At one level NO_ID_ALLOCATED is not part of the String method so it looks outside the Status. Is this fair?

Bill Segall

@billsegall

But then we compare against that in Subscription:channelStatus() and in ClientConductor:ChannelStatus we seem to map NO_ID_ALLOCATED on to ACTIVE.

Thanks for any comments you might make to help me here

Bill Segall

@billsegall

I've got this sorted now I think

Bill Segall

@billsegall

@mjpt777 Is there a simple way for me to run a single Java test and log the archive output? I'd like to run only the replaymergetest and compare the logs of the archiving media-driver against mine.

Dmitry Vyazelenko

@vyazelenko

@billsegall If you want to capture the data stored in the Archive you can make the ReplayMergeTest fail (e.g. modify the test and put fail() or throw exception from it). When the test fails it will capture all of the data (media driver, Archive etc.) and store under the build/test-output directory (i.e. aeron-system-tests/build/test-output/io.aeron.archive.ReplayMergeTest-shouldMergeFromReplayToLive).

The tests usually delete the data when they start so the data is there in the corresponding archiveDir when the test ends. Thus you can simply navigate to that dir to see what is inside.

Dmitry Vyazelenko

@vyazelenko

But if you want to see the Archive log output then you need to enable agent logging. To do this you'll first need to assemble the jars (i.e. ./gradlew clean assemble -x javadoc) and then add the logging options to the test executable. Here is an example which enables admin events for the media driver and all of the events for the Archive:

-javaagent:<path_to_aeron_project>/aeron/aeron-agent/build/libs/aeron-agent-1.38.0-SNAPSHOT.jar
-Daeron.event.log=admin
-Daeron.event.archive.log=all

Bill Segall

@billsegall

Many thanks @vyazelenko

MarshalOfficial

@MarshalOfficial

Hi guys, is there any best practice to handle concurrency on publish side??
assume that we have a client that wants to handle 1 Million concurrent requests, these requests must send to the Aeron server-side via publication offer. now we have a singleton expandable array buffer that overwrites and is corrupted by concurrency.

3 replies

niclar

@niclar

@mjpt777 -Re. the encryption addon, is only the payload encrypted and with what method, does it run in a separate c process encrypting/decrypting the logbuffer, and is source code provided ?

Martin Thompson

@mjpt777

@niclar Please send an email to sales@real-logic.co.uk if you want to know more about the premium extensions.

niclar

@niclar

will do

arkadysh-tr

@arkadysh-tr

Hi,

i am new in Aeron

and Aeron Cluster

arkadysh-tr

@arkadysh-tr

Can you advice some good tutorial/video/course on deep level ? I supposed configure and launch Aeron cluster with 3 nodes ,that will run some FIX client service.I read Cook book and Cluster tutorial ,but i still feel that it is not deep enougth to be able create real product

2 replies

Martin Thompson

@mjpt777

@arkadysh-tr This channel is not for free support. Cluster is a complex topic and if you need help we can offer commercial support. Please contact sales@real-logic.co.uk from your work email address.

Aleksey

@aleksisch

Hi guys, I have publishers/subscribers setup, which works well. Now I want to send messages locally by spy interface, am I right, that all I need is add "aeron-spy:" in the beginning of url subscriber string?

Martin Thompson

@mjpt777

@aleksisch This channel is for contributors. Please ask general questions on Stackoverflow or sign up for commercial support.

Aleksey

@aleksisch

ok, thx

vsadeghi

@vsadeghi

Hi everybody, I'm looking for a solution to log all requests/errors in a log file to parse logs with ELK
Is there any best practice or sample? or what's the Aeron recommendation for this requirement?

simbob

@simbob:matrix.org

[m]

anyone tried to get it running with java 8 on mac arm64?

sunbiaozj

@sunbiaozj

hi,i am new in Aeron

sunbiaozj

@sunbiaozj

if i have three matchines, i want to test aeron by udp multicast type, three matchines in one local network(lan), if pub matchineA ip 192.168.31.100, sub1 matchineB ip 192.168.31.120, sub2 matchineC ip 192.168.31.121, i want to know how config channel for udp multicast, special the "interface" paramter, all subs can receive messages from pub? thank you!

Martin Thompson

@mjpt777

@sunbiaozj This channel is for contributors. Please use StackOverflow for questions.

sunbiaozj

@sunbiaozj

@mjpt777 ok

sunbiaozj

@sunbiaozj

224.0.1.1 is multicast ip

Where communities thrive

People

Repo info

Activity