by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 21:24

    tmontgomery on master

    [C]: add in sample utils and pu… (compare)

  • 16:48

    tmontgomery on master

    [C]: align default channel UDP … (compare)

  • 16:26

    tmontgomery on master

    [C]: add basic subscriber and a… (compare)

  • 15:30

    mjpt777 on master

    [Java] Remove relocate of HdrHi… (compare)

  • 14:36

    tmontgomery on master

    [C]: fix sleep to handle greate… (compare)

  • 14:35
    mjpt777 closed #211
  • 14:35

    mjpt777 on master

    [Java] Allow for wrapping a zer… (compare)

  • 02:04

    tmontgomery on master

    [C]: add counter availability a… (compare)

  • 00:23

    tmontgomery on master

    [C]: add publication is_connect… (compare)

  • 00:07

    tmontgomery on master

    [C]: add basic_publisher app fo… (compare)

  • Jun 01 21:55

    tmontgomery on master

    [C]: add windows getopt equival… (compare)

  • Jun 01 17:25

    mjpt777 on master

    [Java] Formatting. (compare)

  • Jun 01 16:32
    artem-anisimov opened #211
  • Jun 01 14:58

    mjpt777 on master

    [C] Formatting. (compare)

  • May 31 17:06

    mjpt777 on master

    [Java] Include recordingId when… (compare)

  • May 31 15:52

    tmontgomery on master

    [C]: wrire in close handlers an… (compare)

  • May 30 18:22

    tmontgomery on master

    [C]: add subscription to image … (compare)

  • May 30 17:09

    mjpt777 on master

    [Java] Archive logging improvem… (compare)

  • May 30 17:09
    mjpt777 closed #949
  • May 30 16:57
    vyazelenko commented #949
Yonggang Luo
@lygstate
and also I flushed the content in C but still not working.
cause we can not make sure cncByteBuffer with valid file length, only after the version are checked, we have the valid file length
Either way. we need mapping the file twice for retrieve the exact file length
Yonggang Luo
@lygstate
    private static MappedByteBuffer waitForFileMapping(
        final File cncFile, final long deadlineMs, final EpochClock epochClock)
    {
        while (true)
        {
            try (FileChannel fileChannel = FileChannel.open(cncFile.toPath(), READ, WRITE))
            {
                final long fileSize = fileChannel.size();
                if (fileSize < CncFileDescriptor.END_OF_METADATA_OFFSET)
                {
                    fileChannel.close();
                    if (epochClock.time() > deadlineMs)
                    {
                        throw new AeronException("CnC file is created but not populated");
                    }

                    sleep(Configuration.IDLE_SLEEP_MS);
                    continue;
                }

                final MappedByteBuffer mapped = fileChannel.map(READ_WRITE, 0, fileSize);
                fileChannel.close();
                return mapped;
            }
            catch (final IOException ex)
            {
                throw new AeronException("cannot open CnC file", ex);
            }
        }
    }
The fileChannel should be closed, or occasionally the aeron directory can not be removed on windows. may also in other os, can not be sure.
fileChannel should be closed after file mapping
Yonggang Luo
@lygstate
FlushFileBuffers((HANDLE)_get_osfhandle(fd));
Cause the testing procedure to be very slow. That's not we want
Martin Thompson
@mjpt777
@lygstate The try statement will close the channel after mapping. What you are suggesting is not required.
There is a known issue with the C driver on Windows where it cannot be deleted. There is an issue outstanding on it that. As we have said many times we know there are issues with the C driver on Windows and thus it is not supported.
Yonggang Luo
@lygstate
Oh, see that, It's my new code cause the file not closed, not the try one, I've fixed that
I place file length checker right after checkVersion and before the mpsc constructor, the msvc system tests are all working fine.

     /**
@@ -1360,6 +1375,30 @@ public class Aeron implements AutoCloseable

                 CncFileDescriptor.checkVersion(cncVersion);

+                /* make sure the cnc.dat have valid file length before init mpsc */
+                try
+                {
+                    final int computedFileSize = CncFileDescriptor.computeCncFileLengthFromMetaDataBuffer(
+                        cncMetaDataBuffer, Configuration.filePageSize());
+                    final FileChannel fileChannel = waitForFileLength(
+                        cncFile, deadlineMs, epochClock, computedFileSize);
+                    if (this.cncByteBuffer.capacity() != computedFileSize)
+                    {
+                        final MappedByteBuffer cncByteBuffer = this.cncByteBuffer;
+                        this.cncByteBuffer = null;
+                        this.cncMetaDataBuffer = null;
+                        IoUtil.unmap(cncByteBuffer);
+                        this.cncByteBuffer = fileChannel.map(READ_WRITE, 0, computedFileSize);
+                        this.cncMetaDataBuffer = CncFileDescriptor.createMetaDataBuffer(this.cncByteBuffer);
+                    }
+                    fileChannel.close();
+
+                }
+                catch (final IOException ex)
+                {
+                    throw new AeronException("cannot open CnC file", ex);
+                }
+
                 final ManyToOneRingBuffer ringBuffer = new ManyToOneRingBuffer(
                     CncFileDescriptor.createToDriverBuffer(cncByteBuffer, cncMetaDataBuffer));

@@ -1403,7 +1442,7 @@ public class Aeron implements AutoCloseable
             try (FileChannel fileChannel = FileChannel.open(cncFile.toPath(), READ, WRITE))
             {
                 final long fileSize = fileChannel.size();
-                if (fileSize < CncFileDescriptor.CNC_VERSION_FIELD_OFFSET + SIZE_OF_INT)
+                if (fileSize < CncFileDescriptor.END_OF_METADATA_OFFSET)
                 {
                     if (epochClock.time() > deadlineMs)
                     {
@@ -1424,6 +1463,30 @@ public class Aeron implements AutoCloseable
         }
     }
@mjpt777 Are you refer to real-logic/aeron#620
Yonggang Luo
@lygstate
The Java Media Driver attempts the delete the log buffer during the slow tick of the conductor duty cycle and if it fails it increments the FREE_FAILS counter and tries again in the next cycle. The C Media Driver should do similar to support Windows.
Which test is about delete the log buffer, I may have look at that
Yonggang Luo
@lygstate
Todd L. Montgomery
@tmontgomery
that test is not Windows capable yet. It's only UNIX compatible at the moment. And thus why it is not run.
Hristo I Stoyanov
@hrstoyanov
Just curious if customer are using Aeron on Java 8 or Java 11+ mostly? If Java 11+ , and with Gradle 6.4 now out, Aeron could become fully modularized and take advantage of jlink (rather than shadow plugins, etc). Btw, thank you for merging the cluster tutorial .. makes thinks a lot easier to understand!
Todd L. Montgomery
@tmontgomery
Java 8 is pretty common from what I've seen. Java 11 is slower. But 14 is faster.
fh
@JoyJava
May I ask, we found that a message with more than 1500 bytes will result in multiple byteBuffer arrays being received.
Does Aeron have any encapsulated tools for splicing byteBuffer arrays? thx
Todd L. Montgomery
@tmontgomery
@JoyJava check into FragmentAssembler and the samples
Yonggang Luo
@lygstate
 aeron-driver/src/main/c/aeron_network_publication.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/aeron-driver/src/main/c/aeron_network_publication.c b/aeron-driver/src/main/c/aeron_network_publication.c
index 7768d25ff..984924a40 100644
--- a/aeron-driver/src/main/c/aeron_network_publication.c
+++ b/aeron-driver/src/main/c/aeron_network_publication.c
@@ -417,17 +417,18 @@ int aeron_network_publication_send_data(

     if (vlen > 0)
     {
-        if ((result = aeron_send_channel_sendmmsg(publication->endpoint, mmsghdr, (size_t)vlen)) != vlen)
+        result = aeron_send_channel_sendmmsg(publication->endpoint, mmsghdr, (size_t)vlen);
+        if (result == vlen)
         {
-            if (result >= 0)
-            {
-                aeron_counter_increment(publication->short_sends_counter, 1);
-            }
+            aeron_counter_set_ordered(publication->snd_pos_position.value_addr, highest_pos);
+        }
+        else
+        {
+            aeron_counter_increment(publication->short_sends_counter, 1);
         }

         publication->time_of_last_send_or_heartbeat_ns = now_ns;
         publication->track_sender_limits = true;
-        aeron_counter_set_ordered(publication->snd_pos_position.value_addr, highest_pos);
     }
     else if (publication->track_sender_limits && available_window <= 0)
     {
@@ -628,7 +629,7 @@ void aeron_network_publication_on_status_message(
             buffer,
             length,
             addr,
-            *publication->snd_lmt_position.value_addr,
+            aeron_counter_get_volatile(publication->snd_lmt_position.value_addr),
             publication->initial_term_id,
             publication->position_bits_to_shift,
             time_ns));
Does this change make sense?
this is what the Java version doing
Dmitry Vyazelenko
@vyazelenko
@hrstoyanov Aeron cannot be modularized in the JDK 9 sense since it targets JDK 8+
Todd L. Montgomery
@tmontgomery
@lygstate sendmmsg is not how Java works.
thus it is different
also, the second change is an unnecessary volatile load
Pontus Jorgne
@peligroso
Playing around with the Archiver I noticed that multiple sessions/images results in multiple recordings.
My usecase is to have one subsriber reading inputs from multiple producers, archiving the inputs so that I later can replicate all events processed by the subscriber.
This does not seem possible if the producers are exclusive or run in different JVM's. Unless there is a way to merge the resulting recordings to that they are replayed togehter in the same order as they were originally read by the subscriber?
Another possible solution could be to have one proxy- subscriber in the same JVM that will just reshuffle all inputs onto the archived channel/stream. But it seems abit wastefull of system resources.
Perhaps I am tackling this from the wrong angle and would need to rethink my usecase. The end goald is to have a system where each state is replicable.
Todd L. Montgomery
@tmontgomery
the re-publish way is quite possible and fits.... but if you can guarantee that all publishers will publish the same exact stream, then merging them can be done via multi-destinations on the subscriber side and merged into a single image....
but... can you guarantee that each publisher will send the same stream?
Pontus Jorgne
@peligroso
The idea is that each publisher will publish to the same endpoint/stream. I will do some tests on Subscription-side multi-destination. If that would result in multiple publication being merged into one a single image I think it would be exactly what I'm looking for
Todd L. Montgomery
@tmontgomery
well, it only merges them IF the data is the same from both publications in the same positions. If that is not what you are doing, it will not work.
Pontus Jorgne
@peligroso
Ah, gottcha.. ok, then the re-publish seems like my best bet. Thanks
Todd L. Montgomery
@tmontgomery
no worries
Martin Thompson
@mjpt777
@peligroso Re publish on IPC and record that while consuming it in your subscriber.
Pontus Jorgne
@peligroso
Thanks for confirming. You guys are the best!
Todd L. Montgomery
@tmontgomery
Thanks!
volgad
@volgad
Hi all, I am using a mix of C# server on the server side and a scala/jvm client to communicate with it. On the jvm side, I am using akka / akka cluster and wanted to move from tcp / jackson to udp / aeron. For this I believe that I need to use aeron 1.27.0 on the akka side but the latest release I see on https://github.com/AdaptiveConsulting/Aeron.NET is 1.21.2. Has anybody faced similar issue / knows if there is a way of using a different version on each side. At the moment I am getting an error regarding cnc.dat because of the different versions (version not understood: version=15). Any reference or pointer is appreciated. Thanks a lot
James Watson
@JPWatson
It’s not possible to use a client with a different cnc protocol version to the driver.
I plan to update the .NET version to 1.27.0 shortly which should solve your problem.
Martin Thompson
@mjpt777
@volgad Later versions support semantic versioning so when @JPWatson updates then things should get easier going forward.
volgad
@volgad
Sounds good. Thank you both
James Watson
@JPWatson
Has anybody been able to work around https://youtrack.jetbrains.com/issue/IDEA-237290 in the latest IntelliJ?
Michael Barker
@mikeb01
Yes, I have to manually add the dependency to Agrona on the 2 generated-src modules (one in archive and one in cluster).
James Watson
@JPWatson
👌🏼 Good tip. Thanks!
StevenCurran
@StevenCurran
Hey guys, is this the same issue for agona?
trying to build master on intellij 2020.1.2 and org.agrona.generation.DoNotSub; cannot be resolved
Todd L. Montgomery
@tmontgomery
not having an issue with Agrona and IntelliJ 2020.1.1, so not sure.
Martin Thompson
@mjpt777
@StevenCurran Yes the issue is the same with the generated code modules. It needs a dependency on agrona.
I've been chasing with Jetbrains to have this fixed.
zagorulkinde
@zagorulkinde
Hi folks, Did you try to test aeron on latest java version? (14.X) What jdk vendor do you use for testing?
Martin Thompson
@mjpt777
@zagorulkinde We run CI on Java 8 and 11. I have tried it on 14 and it works well. It does not build on 14 because of the Shadow plugin uses an old version of ASM but this is beyond our control. Aeron over UDP is faster on Java 14 compared to 8 or 11. 11 is the slowest.
zagorulkinde
@zagorulkinde
Thanks Martin
ratcash
@ratcashdev
Hi all! I have just hit the ctx.maxConcurrentSessions() limit and I can't help but think it's there almost only for you to watch out for scared potential clients reporting this exception - knowing that he might have a reasonably utilized cluster to offer him consultancy :D
I mean apart from the fact that it seems to be there to keep the sessionIdMap small, and in cache, isn't it?
Todd L. Montgomery
@tmontgomery
Aeron 1.28.2 pushed to maven central