real-logic/Aeron

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

Michael Barker

@mikeb01

Hi @jmazar we've changed the role of this room to be specific to public contributor discussions. For support-like queries the best place is: https://stackoverflow.com/questions/tagged/aeron

dantaras

@dantaras

Hi guys! Not sure if this is the right place to post, let me know if not. I'm working on a Julia version of an Aeron client using the C++ client as a model and had a question about the position limit mechanism.

In the offer methods in https://github.com/real-logic/aeron/blob/master/aeron-client/src/main/cpp/Publication.h position is checked against a limit (eg: https://github.com/real-logic/aeron/blob/master/aeron-client/src/main/cpp/Publication.h#L358) and if smaller publishing can continue. Shouldn't that check take into account how much data there is to be published? If, for example, the position limit is higher by one than the current position and we want to publish 100 bytes, then we will be allowed to publish, go over the limit and in extreme cases we might overwrite data which subscribers haven't read.

Michael Barker

@mikeb01

@dantaras This is handled within the TermAppender (TermAppender.h:164). If you are writing a client for another language, I would recommend looking at the C++ wrapper over the C API. There is quite a lot of complexity in the API and writers of clients for different languages have found it difficult to keep up with feature changes. Hence we've created the C API implementation to enable the development of APIs in other languages more quickly.

dantaras

@dantaras

Thanks for the response, Michael!

What is the link between position limit here https://github.com/real-logic/aeron/blob/master/aeron-client/src/main/cpp/Publication.h#L344 and term length here https://github.com/real-logic/aeron/blob/master/aeron-client/src/main/cpp/concurrent/logbuffer/TermAppender.h#L158? I was under the impression that position limit can be different and could limit publication within the same term.

I will have a look at the C API as well. Does the shared memory driver API change often?

Michael Barker

@mikeb01

@dantaras data won't be overwritten as it will only allow that message if it fits within a term and there is another cleaned term between the one being published to and the one being read from. If message is larger than the remaining space in the term then the term will be padded, the offer rejected with ADMIN_ACTION and the position will then be higher (or equal) to the limit and offer will be blocked with back pressure.

dantaras

@dantaras

I get it now, thanks @mikeb01 !

Valto Vipul

@ValtoVipul_twitter

hi guys, is there a way to fall back to the previous mode of segment file naming convention? 10-0,10-1,10-2, etc?

I have some scripts which used that so would like to see if possible to use the old behaviour

oops it seems no more help in this room

michaelredman

@michaelredman

Hi, I am trying to persist the egress with aeron cluster, the clustered driver already has an archive and various channels setup, is it a bad idea to use ClusteredMediaDriver.archive().startRecording(my_log) and try connect to it over udp? will the cluster client get confused?

Martin Thompson

@mjpt777

@michaelredman This channel is for contributor discussions only. If you require support we have commercial offerings.

William

@ilove7412369_twitter

I believe i encounter an Aeron Bug.
i publish multidestintaion cast from Host A to HostA and HostB, big message as big as 300k bytes. at rate say 10 times per second.
After 1-2 hours, one of the host , say host B, will get corrupted message. We detect that as it is SBE message with wrong data. While the same message properly processed correctly and successfully.
And then, we will not able to receive any message in host B.
I think bottom line we got exception instead of corrupt message as that make me worry the integrity of the data via aeron.

Martin Thompson

@mjpt777

@ilove7412369_twitter This channel is not for support. If you can create a test that reproduces what you are seeing then raise an issue.

@ilove7412369_twitter When we have investigated issues of data corruption in the past it has almost always been a bug in the application. It is possible it is a bug in Aeron but without a reproducible test we cannot investigate without a support contract.

gorive

@gorive

Just stumbled across Aeron- got a question

1) where does the actual udp transport happen - where it sends the messages to all clients ? Is that in the media driver ?

2) lets say i use it as a fast message broker on the internet and only as unicast - how many clients ( clients connecting to the udp port on the server ) from the outside can Aeron handle without degrading in speed ? Are there any benchmarks on the pure transport level if we want to burst out alot of packets to be distributed to many clients ?

Michael Barker

@mikeb01

1) Yes. 2) This is complicated question and there are numerous tuning options and approaches that can affect this. We can offer consulting in this area: https://real-logic.co.uk/contact.html.

particulitis_julien

@particulitis_twitter

Quick question, I'm a bit confused. The cookbook mentions using ReservedValueSupplier as a checksum. So I'm assuming that Aeron doesn't do a checksum on the receiver messages ? I don't see any official mentions of that except in the wiki

Martin Thompson

@mjpt777

@particulitis_twitter UDP provides its own checksum. The reserved value is for additional data carriage which the application can apply. In future please ask questions on Stackoverflow, this group is for contributor discussions.

particulitis_julien

@particulitis_twitter

Ok, sorry about that, and thanks for the quick reply Martin :)

eduneeraj2019

@eduneeraj2019

morning

eduneeraj2019

@eduneeraj2019

Can aeron connect between multiple switch and multiple subnet? Its works fine on same switch

Martin Thompson

@mjpt777

@eduneeraj2019 This channel is not for support questions.

eduneeraj2019

@eduneeraj2019

Tks @mjpt777 where shud i ask support qn ?

Martin Thompson

@mjpt777

You can approach us for a commercial support contract or try Stackoverflow.

chertar

@chertar

I'm looking to get started with SBE. Is there a tutorial or example to learn from? I've been searching and can't find anything

A "hello world" type example would be perfect

Actually, never mind, I found one. https://github.com/real-logic/simple-binary-encoding/tree/master/sbe-samples/src/main/java/uk/co/real_logic/sbe/examples

Ronen

@ronenhamias

Hello everyone is there somewhere a simple benchmark for aeron-cluster that i can learn from?

Michael Barker

@mikeb01

Hi @ronenhamias, the focus of this channel is community contribution discussion. Support questions should come through a commercial contract or via stackoverflow.

We have a cluster benchmark here: https://github.com/real-logic/benchmarks/blob/master/benchmarks-aeron/src/test/java/uk/co/real_logic/benchmarks/aeron/remote/ClusterTest.java.

Ronen

@ronenhamias

@mikeb01 thank you very much!

Bill Segall

@billsegall

@mjpt777 I'm probably being blind and not seeing it but is there any documentation on the aeron-archive security model. I'm looking to implement this in aeron-go and I've been looking at the java code. I'm not sure I really comprehend the model for either connection establishment and/or message delivery.

Todd L. Montgomery

@tmontgomery

@billsegall the archive uses the cluster model for Authenticator, etc.

specifically, there is some in https://github.com/real-logic/aeron/blob/master/aeron-cluster/src/main/resources/cluster/aeron-cluster-codecs.xml#L97

Bill Segall

@billsegall

Lovely! So this is connection auth. Am I right there is no mechanism for selective delivery of messages so this is all I need to worry about?

Todd L. Montgomery

@tmontgomery

yes, it is on the connection.

Bill Segall

@billsegall

Thanks!

Todd L. Montgomery

@tmontgomery

no worries

supergaga

@supergaga

Can I subscribe 2 different UDP channel in one aeronCluster Client?

Michael Barker

@mikeb01

Hi @supergaga we are using this chat room is for public contributor discussions. For support queries you can post your question to stack overflow (https://stackoverflow.com/questions/tagged/aeron) or contact Real Logic for a commercial support contract.

supergaga

@supergaga

@mikeb01 okay . thanks reply

William

@ilove7412369_twitter

@keithwong1 ConcurrentPublication is thread safe; ExclusivePublication is not thread safe. ConcurrentPublication is returned from Aeron.addPublication. ThreadingMode for the driver is an independent feature for configuring thread usage by the driver.

But is ConcurrentPublication safe to use when multiple "processes" publish to same endpoint?

Martin Thompson

@mjpt777

@ilove7412369_twitter We do not offer support in this channel. Please ask questions on Stackoverflow.

William

@ilove7412369_twitter

I thought my level of question is the same level as Keithwong1, so is that Keithwong1 has paid support so can ask it here?

Martin Thompson

@mjpt777

We have a support mailing list for those on commercial support.

Historically we have given support on this channel and people have taken advantage.

KabDeveloper

@KabDeveloper

Caused by: java.lang.IllegalStateException: Missing file for cnc: c:\datapath\cnc

Someone can help with this error?

Michael Barker

@mikeb01

@KabDeveloper We have switched to using this channel for public contribution discussion. For support queries you can post your question to stack overflow (https://stackoverflow.com/questions/tagged/aeron) or contact Real Logic for a commercial support contract.

weltam

@welly87

hi all, i try posting on stackoverflow but seems it not the appropriate place for this questions it's getting closed :D

Previously i read in docs that C Media Driver currently still on going development but doesn't support clustering like Java does. Is this still the case today? I try to look at source code but seems it's no clustering related code there. Not sure if i'm looking at right branch.

Is it a goal for C Media Driver to be in par with Java Media Driver? And which direction it will go?

thanks a lot

Where communities thrive

People

Repo info

Activity