a protocol with zero overhead and powerful abstractions -- https://zenoh.io
amazing what a difference --release makes here for those zn_pub/sub_thr programs
30k msgs/s in debug, 700k msgs/s in release
some major compiler voodoo?
everything inlined into a single call? like wow
@bfrog, FYI, we are about to merge a change that should further improve the performances for small data. To make a long story short we found out that Rust async futures tend to bloat the stack and as a consequence drop performances as deep the async call stack becomes. We’ve done some refactoring to keep the stack size small and early measurements are extremely promising. The merge should happen sometimes toward the end of next week. Thus stay tuned.
For what concerns the debug vs release our flags are set to make sure that debug compiles faster while release does global optimisation and cpu specific code generation.
nice, yeah I know the nats client folks (same author as sled database!) actually dropped using async as they found it didn't give them the benefits they were looking for
that's different though... single socket client really
would be interesting to see a nats comparison as well
We ended-up just constraining the async portion. If you are curious can take a look this branch
We’ve done measurement with NATS in the past, we’ll do again once we merge these changes. Two things to keep in mind. With NATS you have to expliticely flush from the user API. Zenoh does the batching automatically. Additionally our protocol has some higher level primitives, thus our routing code is a bit more complicated…. If you consider all of that, our performance are not so bad. But I am really looking forward to see where we get with the new async/sync split.
is it possible to use tokio instead of async-std?
seems like most things are pretty tied into async-std
async-std runtime is capable of coexsit with tokio… in fact some network-specific aspects (like the QUIC implementation) leverage tokio
Note that if you use the zenoh router and want to access its REST API through the NAT, you’ll also need to open TCP/8000
thanks @JEnoch , I forgot about that one
Hi guys, a question about minimal overhead of a Zenoh data message. On zenoh.io it is stated that "the minimal wire overhead of a data message is 4 bytes.". is this actually possible? I looked at some zenoh data messages in Wireshark, using the dissector, from which I calculated an overhead of 13 bytes. (I used the python API to publish something) Are the 4 bytes maybe just for raw zenoh.net usage? (leaving out stuff like encoding, etc.) Especially I wonder how this is solved with the SN field, because it alone takes 4 bytes in the frame.
Hi @OliLay , 4 bytes is in fact the minimal overhead achievable in zenoh data messages, however this requires a special configuration.
As you correctly identified, the SN filed takes more bytes… let me explain briefly how it works:
A SN is encoded as variable-length (VLE). That means that the footprint on the wire depends on its value. As you can imagine a 64 bit integer always takes 8 bytes when encoded as is. Instead, while encoded as VLE the size on the wire depends on its value. E.g., encoding the value 0 always takes 1 byte on the wire (event if it is a 64 bit integer).
To limit the number of bytes the SN can take, two zenoh endpoints can negotiate its resolution at session establishment. As per VLE works, an SN resolution of 128 will allow to always stick to 1 byte max.
By default, zenoh uses 4 bytes resolution for the SN. This is to cope with high throughput. Indeed, larger the SN window, the more messages you can have on the fly so as to increas throughput. So, in short limiting the SN resolution is very beneficial in those cases when operating over constrained networks at low throughput.
But this is only one piece of the story. Configuring the SN in this way allows to limit the wire overhead to 2 bytes. So, where are the other 2 bytes?
This is in the data message itself and how resources are represented:
In zenoh each resource is represented by a key, in the form of URI. E.g. /home/sensor/temperature
The key can be arbitrary long. This is very handy for the user who is completely free to define his own key structure, as complex/nested as he wants to be.
However, transmitting the whole key is not wire efficient at all. For this reason, in zenoh we have a mechanism to do some dynamic mapping between the complete string form of the key into a more compact integer form, called simply resource ID. This means that we have a wire-efficent mapping for identifying resources departing from simple integers sent on the wire. This resource ID is encoded as VLE, thus the possibility to use only 1 byte.
In order to use this wire-efficient representation you may need to use the declare_resource() primitive in zenoh, which does exactly that under the hood.
So, summarizing. 4 bytes are the minimal overhead that can be achieved by zenoh thanks to: 1) SN negotiation and limited to 1 byte. This might have an impact on the achievable throughput due to max num of messages that can be on the fly at the same time. 2) The efficient mapping between string and integer resources on the wire.
with that sync branch, I get 1.5 million msgs/sec with the shm thr samples, amazing
so... yeah I think removing some async stuff there really did do some magic
Yes, that’s what we discovered by investigating in depth the async stack. It’s very powerful, but has a huge cost when used everywhere. So, the sync branch is really an ongoing effort of fencing the async part in few specific parts in zenoh.
Hi @Mallets. Thank you very much for the information, which makes the protocol much clearer for me now. I was a bit confused, because I have zenoh TCP traffic captured, and as I saw in the source code that requires the frame length and the payload length to be prepended each. With leaving that aside (e.g. using UDP) and using a 1 byte SN as you said, I get how you can have a 4 byte data message on the wire. Thanks again :)