with that sync branch, I get 1.5 million msgs/sec with the shm thr samples, amazing
so... yeah I think removing some async stuff there really did do some magic
Hi @Mallets. Thank you very much for the information, which makes the protocol much clearer for me now. I was a bit confused, because I have zenoh TCP traffic captured, and as I saw in the source code that requires the frame length and the payload length to be prepended each. With leaving that aside (e.g. using UDP) and using a 1 byte SN as you said, I get how you can have a 4 byte data message on the wire. Thanks again :)
kydos (Angelo Corsaro) for my message size avg of about 512 bytes, shm does about 100-200k msgs/s more
you know, on my desktop, not necessarily a great comparison given a lot of these things are likely to be small embedded linux arm devices, still impressive to me
@OliLay yes, right. In case of operating over TCP we need to add 2 bytes to identify the actual batch length on the wire. This is not needed on UDP or e.g. QUIC. Moreover, to complete really the story, the 2 bytes for TCP + 2 bytes for SN number are sent only once per wire-frame. Each frame then can contain multiple data messages. E.g., considering a frame batch is max 64KB, you can fit in it thousands of 8 bytes tiny messages. For each of those message you get the minimal 2 bytes overhead for data messages. But then, the 2 bytes of TCP and 2 bytes of the SN are sent only once for all those thousands of tiny messages.
also is nats-bench somehow cheating? seems like a peer to peer connection would be significantly faster than one with a broker involved
nats-bench is just a hair behind this thing
which... is surprising, given I imagine at least a few additional copies and syscalls are involved
Ok, I see that @JEnoch replied before me :D
Tbh we didn’t have a look into nats-bench deeply, so I don’t really know about the inner details… what do you experience?
ah ok that makes sense, I figured it'd just take the ip addr but it didn't like that
nats-bench pushing 512 byte messages to a local nats server, with a single publisher/subscriber gets me 1.4 million msgs/s
I'd rather see latency histograms than throughput though to be honest
@halfbit:matrix.org, what are you using for benchmarking NATS, dds-bench? If so this is not really apple to apple comparison, please take a look at this line https://github.com/nats-io/nats.go/blob/3b1f6fcc1e1014c838036367494c3012523166b0/examples/nats-bench/main.go#L168
they have their own bench tool which I mean, yeah, hard to say its doing even remotely the same stuff
Our scheduling is done at the transport level, thus it is transparent for the user.
its interesting that batching is even needed, I get the ideal scenario is avoiding too many sys calls... would io_uring do better without having to batch up messages?
push messages into a queue with the kernel, let it do things as fast as it can kind of deal
rather than this goofy, lets buffer up a million messages then call write once
that nats-bench is doing at least
At a transport level zenoh sends frames, a frame can include multiple messages. One frame is sent with a single sys-call (if we do not need to fragment), how we pack it up is part of our scheduler, but the goal is taking care of reducing syscalls while at the same times ensuring message prioritisation.
To be more precise within a zenoh session we can use multiples links, e.g. TCP/IP connections, to ensure that we can better recover congestions over WAN — that is a relatively well known trick.
zenoh seems to be capped by the line rate between two computers, which is fantastic
at least on this puny gigabit network I have
no 10g to test against :-)
On a native PC, I launch:./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.89.76.138:7447 -e tcp/10.89.76.139:7447
On a container running on another PC, I launch:./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.92.70.153:7447 -e tcp/10.89.76.138:7447
The container's host is forwarding port tcp/7447 into the container. I confirmed that port forwarding is working with iperf.
However when I run a ROS 2 publisher on the native PC, and then list topics in the container, I don't see the published topic.
I can see the two instances of
zenoh-bridge-dds talking to each other in wireshark
That's a wireshark capture from the host
Interestingly, in captured packet 19 I can see the IP address of the container (10.92.70.153) being sent to the native PC (10.89.76.138). I don't know how zenoh discovery works but is it possible that the instance of zenoh running inside the container is telling the instance running on the native PC to connect to the container's IP rather than the IP of the host PC (which would then NAT and forward that connection into the container)?
Is there any documentation on what configuration parameters can be set in a configuration file?
It's working using a router on the host PC. I can communicate over ROS between the native PC and the container :)
I'd like to get it going over NAT without the router though, if that's possible.
Interesting... if I stop publishing the topic, it's still listed as available. I guess that
zenoh-bridge-dds keeps making the topic available even though there are no publishers for it anymore.