Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 14:53

    mjpt777 on master

    Tidy up after merge of PR #257. (compare)

  • May 22 16:08

    vyazelenko on master

    [Java] Set min Agrona version t… (compare)

  • May 20 04:26

    mikeb01 on master

    [Java] Use a message to ensure … (compare)

  • May 20 01:58

    mikeb01 on master

    [CMake] Remove debug for releas… (compare)

  • May 19 10:21
    m-anthony commented #258
  • May 18 14:00
    sunbiaozj commented #1321
  • May 18 13:12
    mjpt777 labeled #258
  • May 18 13:12
    mjpt777 commented #258
  • May 18 13:09
    m-anthony edited #258
  • May 18 11:41
    m-anthony opened #258
  • May 18 05:18
    mikeb01 synchronize #1272
  • May 17 22:27

    mikeb01 on 1.38.x

    Reduce sharing on image endpoin… (compare)

  • May 17 21:40

    mikeb01 on master

    Reduce sharing on image endpoin… (compare)

  • May 17 21:40
    mikeb01 closed #1322
  • May 17 21:14
    tmontgomery review_requested #1322
  • May 17 19:59

    mjpt777 on master

    [Java] Update recordingId on an… Merge remote-tracking branch 'o… (compare)

  • May 17 15:01

    mjpt777 on master

    [Java] When attempting ClientSe… (compare)

  • May 17 15:01
    mjpt777 closed #1323
  • May 17 14:56
    tmontgomery commented #1323
  • May 17 14:53
    tmontgomery closed #1321
Michael Barker
@mikeb01
@NikitaMishin As mentioned, if you could ask support questions on Stack Overflow and tag it with "aeron". You are more likely to get a response there.
Vitaly
@rvit34
ratcash
@ratcashdev
Hello gents, could you please help with the following scenario regarding cluster election? Assuming a leader dies with appendPositionL and commitPositionL. Next one of the members becomes a candidate while having commitPositionM1 < commitPositionL < appendPositionM1 < appendPositionL. So i was wondering what happens when he gets (eventually) elected?
a) he truncates his log to commitPositionM1 (just like all the followers do onNewLeadershipTerm ),
b) he keeps the messages in his log (beyond the commit position) and eventually distributes these to members and form a consensus
c) nothing is guaranteed during elections, and he may or may not retain those log entries beyond the commit position.
thanks in advance.
Martin Thompson
@mjpt777
@ratcashdev You know we don't offer support via this channel. Please consider signing up to commercial support. You are long over due ;-)
ratcash
@ratcashdev
Yes! I am. You're right, sorry. You made me smile big time, btw, so thanks for that.
Chris Jones
@chrisejones

Hi, we have a multi-socket box and have two processes using Aeron IPC on a different numa node to where the media driver is running. The IPC term buffers are ending up on the media driver node according to /proc/<pid>/numa_maps.

Do we need a separate media driver on the other socket to avoid this?

Martin Thompson
@mjpt777
@chrisejones Please ask general questions on Stackoverflow.
Chris Jones
@chrisejones
sure, thanks
nemanjalucicfidexx
@nemanjalucicfidexx
Hello! SBE C++ generated code changes field names to upper case for methods like put, get and skip (e.g. for field "payload" C++ class contains "putPayload" method). It would greatly benefit our generic code if the field name format could be controllable using cmd arguments. Even better if we could switch between camel case and snake case. We played a bit with SBE code and managed to modify it accordingly. We a contribution like this from our side be something that you would consider?
Martin Thompson
@mjpt777
@nemanjalucicfidexx This is not the place for SBE discussions. Please use the SBE repo if you wish to have a PR considered.
nemanjalucicfidexx
@nemanjalucicfidexx
Oh, sure. Thanks!
Michael Barker
@mikeb01
@denisxor This channel is for community contributions. Please post support questions to https://stackoverflow.com/questions/tagged/aeron or contact Real Logic Ltd for a commercial support contract.
denisxor
@denisxor
@mikeb01 thanks. I'll go to stackoverflow.
Doru-Mihai Muntean
@DoruM

Hello,

What is the coding style of Aeron based on? I remember that there was a keynote lecture or something. Can someone please point me to it?

4 replies
Nikita
@NikitaMishin

Hello!

i have a quick question regarding benchmarks
found weird downside when testing on two remote machines:

Subscrbier:
java -cp aeron-all/build/libs/aeron-all-1.34.0.jar -Daeron.sample.embeddedMediaDriver=true -Daeron.sample.channel="aeron:udp?endpoint=0.0.0.0:20006" io.aeron.samples.RateSubscriber
Publisher
java -cp aeron-all/build/libs/aeron-all-1.34.0.jar -Daeron.sample.embeddedMediaDriver=true -Daeron.sample.channel="aeron:udp?endpoint=34.245.39.15:20006" io.aeron.samples.StreamingPublisher

on both machines i adjust $ sudo sysctl net.core.rmem_max=2097152
$ sudo sysctl net.core.wmem_max=2097152 as mentioned in your wiki page.

this is small snapshot:
6336 msgs/sec, 2.027e+05 payload bytes/sec, totals 57032 messages 1 MB
6247 msgs/sec, 1.999e+05 payload bytes/sec, totals 63284 messages 1 MB
6255 msgs/sec, 2.001e+05 payload bytes/sec, totals 69544 messages 2 MB
6288 msgs/sec, 2.012e+05 payload bytes/sec, totals 75838 messages 2 MB
6266 msgs/sec, 2.005e+05 payload bytes/sec, totals 82110 messages 2 MB
6270 msgs/sec, 2.006e+05 payload bytes/sec, totals 88384 messages 2 MB
6269 msgs/sec, 2.006e+05 payload bytes/sec, totals 94658 messages 2 MB
6279 msgs/sec, 2.009e+05 payload bytes/sec, totals 100942 messages 3 MB
6291 msgs/sec, 2.013e+05 payload bytes/sec, totals 107238 messages 3 MB
6292 msgs/sec, 2.013e+05 payload bytes/sec, totals 113534 messages 3 MB
6279 msgs/sec, 2.009e+05 payload bytes/sec, totals 119818 messages 3 MB

Am i missed smth? Do i need to setup specific configuration to run your benchmarks

because for same setup when launched on same machine for udp i get
1.047e+06 msgs/sec, 3.351e+07 payload bytes/sec, totals 1048218 messages 31 MB
8.934e+06 msgs/sec, 2.859e+08 payload bytes/sec, totals 10000000 messages 305 MB

Martin Thompson
@mjpt777
We don't provide support or answer general questions in this channel.
John Davis
@unicomp21
Has anyone taken a look at Seastar and co_await? For Aeron? The redpanda kafka replacement is benefiting greatly from it.
Michael Szymczak
@michaelszymczak
Hi, I have noticed some interesting and potentially dangerous use case in how Artio treats numeric constants such as ZERO. I did not know what to make of it, so I created an issue for that real-logic/artio#452
1 reply
Vachagan Balayan
@vach_b_twitter
Hi Everyone, i'm having some hard time with aeron cluster, are there any books resources that could take alook before buggin you here?
i'm reworking a core services for an exchange (every essential service + matching engine) to achieve insane speeds and TPS,
however this one is crypto exchange and they cant just restart things over weekend like banks, so i'm trying to dive into Raft via aeron
but there are some things i cant see how to control and configure...
Michael Barker
@mikeb01
Hi @vach_b_twitter, we don't do Aeron support on this channel (it is for community contribution discussions). You can contact Real Logic to arrange support and consulting (http://real-logic.co.uk). Or you have specific questions on Stack Overflow (using the aeron tag). There is also a tutorial available: https://github.com/real-logic/aeron/wiki/Cluster-Tutorial.
Vachagan Balayan
@vach_b_twitter
thanks, sent an email )
zaradai
@zaradai
I have a gripe with the generated c SBE code for the template id. As an example it generated
SBE_ONE_DEF uint16_t events_instrumentLoad_sbe_template_id(void)
{
return (uint16_t)16;
}
this isn't compile time constant so i am unable to a switch in a fragment handler. I would prefer a simple switch to handle different incoming events vs a hash lookup, am i missing something?
zaradai
@zaradai
just noticed comment above about SBE discussions, please ignore I'll go to SBE.
Niel de Wet
@nieldw

With no changes to our code, but just a restart, we are suddenly facing an unusual ERROR when the Consensus Module is starting up. What could be possible causes? Full stack trace in thread.

ERROR io.aeron.cluster.ConsensusModuleAgent java.lang.IllegalStateException: Cannot assign long arg5 to int

2 replies
Martin Thompson
@mjpt777
@MarshalOfficial This channel is for contribution discussions and not for free support.
Bill Segall
@billsegall

Hi @mjpt777

It's been a while but I'm now looking at adding MDC support to aeron-go.

I was wondering if there is an XML spec for the flyweights used for AddDestination etc? I had a bit of a browse in the aeron git and wiki but failed to find anything.

It all seems pretty transparent from the code - it's just I prefer to start from a spec where possible.

Thanks,
Bill.

Todd L. Montgomery
@tmontgomery
Hi Bill. No spec other than the Javadoc and code for the messages between the clients and driver.
Bill Segall
@billsegall
cool. Thanks!
Todd L. Montgomery
@tmontgomery
if you do have questions, let us know. It's fairly simple and (I hope) straight forward, but more than happy to confirm/deny anything we can.
Konstantin Dmitriev
@konstantind23_gitlab

Is using ReplayMerge necessary for merging replay with live stream? I guess otherwise it is not trivial to find the
cut off point at which subscriber should consume live stream instead of replayed and guarantee "at most once" delivery.

The problem I am facing is that ReplayMerge does not seem to work where live messages are published and recorded over multicast. I am getting error message "channel does not allow manual control" which I guess means ReplayMerge is tring to add subscription to the publisher which publishes over multicast channel that does not support manual control.

All ReplayMerge examples/tests use MDC so I wonder if multicast is supported for this use case

Todd L. Montgomery
@tmontgomery
@konstantind23_gitlab ReplayMerge supports multicast on the liveDestination. The subscription must be a manual one for MDC and multicast.
Konstantin Dmitriev
@konstantind23_gitlab

Thank you, I can get ReplayMerge working now for single publication sessionId. Unfortunately it does not seem to support multiple simultaneous sessions. I have multiple clients publishing to the same streamId over the same multicast channel, each having different sessionId. I tried running one ReplayMerge per session (getting sessionIds from archive recordings) with segregated replayDestinations but ReplayMerge gets confused when attempting to switch to live as it is trying to merge replay that delivers messages from a single session with live stream that delivers events from multiple sessions.

Ideally I would have one recording per streamId instead of the current recording per sessionId. Forcing the shared sessionId on multiple publishers results in archive/subscription treating multiple publications as one session, i.e. removing what it thinks are duplicate message.

Todd L. Montgomery
@tmontgomery
To do what you want, you would need to use session specific subscriptions.
@konstantind23_gitlab This channel is for contribution discussions and not for free support.
Bill Segall
@billsegall
Hi again, I'm a little confused and was hoping to get soem clarity regarding the ChannelEndpointStatus as I'm needing to implement this the golang ReplayMerge.
We seem to have 5 states defined, INITIALIZING = 0, ERRORED = -1, ACTIVE = 1, CLOSING = 2. and then curiously NO_ID_ALLOCATED which is also -1 but an int rather than a long.
At one level NO_ID_ALLOCATED is not part of the String method so it looks outside the Status. Is this fair?
Bill Segall
@billsegall
But then we compare against that in Subscription:channelStatus() and in ClientConductor:ChannelStatus we seem to map NO_ID_ALLOCATED on to ACTIVE.
Thanks for any comments you might make to help me here
Bill Segall
@billsegall
I've got this sorted now I think
Bill Segall
@billsegall
@mjpt777 Is there a simple way for me to run a single Java test and log the archive output? I'd like to run only the replaymergetest and compare the logs of the archiving media-driver against mine.
Dmitry Vyazelenko
@vyazelenko
@billsegall If you want to capture the data stored in the Archive you can make the ReplayMergeTest fail (e.g. modify the test and put fail() or throw exception from it). When the test fails it will capture all of the data (media driver, Archive etc.) and store under the build/test-output directory (i.e. aeron-system-tests/build/test-output/io.aeron.archive.ReplayMergeTest-shouldMergeFromReplayToLive).
The tests usually delete the data when they start so the data is there in the corresponding archiveDir when the test ends. Thus you can simply navigate to that dir to see what is inside.
Dmitry Vyazelenko
@vyazelenko
But if you want to see the Archive log output then you need to enable agent logging. To do this you'll first need to assemble the jars (i.e. ./gradlew clean assemble -x javadoc) and then add the logging options to the test executable. Here is an example which enables admin events for the media driver and all of the events for the Archive:
-javaagent:<path_to_aeron_project>/aeron/aeron-agent/build/libs/aeron-agent-1.38.0-SNAPSHOT.jar
-Daeron.event.log=admin
-Daeron.event.archive.log=all
Bill Segall
@billsegall
Many thanks @vyazelenko
MarshalOfficial
@MarshalOfficial
Hi guys, is there any best practice to handle concurrency on publish side??
assume that we have a client that wants to handle 1 Million concurrent requests, these requests must send to the Aeron server-side via publication offer. now we have a singleton expandable array buffer that overwrites and is corrupted by concurrency.
3 replies
niclar
@niclar
@mjpt777 -Re. the encryption addon, is only the payload encrypted and with what method, does it run in a separate c process encrypting/decrypting the logbuffer, and is source code provided ?
Martin Thompson
@mjpt777
@niclar Please send an email to sales@real-logic.co.uk if you want to know more about the premium extensions.