Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Luca Cominardi
    @Mallets
    no no, zenoh is multi-protocol… by default uses TCP for communication but it can also operate on top of UDP (best effort only), QUIC, TLS and unixsockets
    Geoff
    @geoff_or:matrix.org
    [m]
    OK. I guess if it's using TCP by default then that's fine
    Luca Cominardi
    @Mallets
    by default goes over plain TCP
    Geoff
    @geoff_or:matrix.org
    [m]
    it
    it'll be more reliable going through the NAT
    Luca Cominardi
    @Mallets
    in general yes
    Geoff
    @geoff_or:matrix.org
    [m]
    Thanks! I'll give it a go
    Luca Cominardi
    @Mallets
    Great, let us know if you have any problem!
    Julien Enoch
    @JEnoch
    Note that if you use the zenoh router and want to access its REST API through the NAT, you’ll also need to open TCP/8000
    Luca Cominardi
    @Mallets
    thanks @JEnoch , I forgot about that one
    Oliver Layer
    @OliLay
    Hi guys, a question about minimal overhead of a Zenoh data message. On zenoh.io it is stated that "the minimal wire overhead of a data message is 4 bytes.".
    is this actually possible? I looked at some zenoh data messages in Wireshark, using the dissector, from which I calculated an overhead of 13 bytes. (I used the python API to publish something)
    Are the 4 bytes maybe just for raw zenoh.net usage? (leaving out stuff like encoding, etc.)
    Especially I wonder how this is solved with the SN field, because it alone takes 4 bytes in the frame.
    Luca Cominardi
    @Mallets
    Hi @OliLay , 4 bytes is in fact the minimal overhead achievable in zenoh data messages, however this requires a special configuration.
    Luca Cominardi
    @Mallets
    As you correctly identified, the SN filed takes more bytes… let me explain briefly how it works:
    • A SN is encoded as variable-length (VLE). That means that the footprint on the wire depends on its value. As you can imagine a 64 bit integer always takes 8 bytes when encoded as is. Instead, while encoded as VLE the size on the wire depends on its value. E.g., encoding the value 0 always takes 1 byte on the wire (event if it is a 64 bit integer).
    • To limit the number of bytes the SN can take, two zenoh endpoints can negotiate its resolution at session establishment. As per VLE works, an SN resolution of 128 will allow to always stick to 1 byte max.
    By default, zenoh uses 4 bytes resolution for the SN. This is to cope with high throughput. Indeed, larger the SN window, the more messages you can have on the fly so as to increas throughput. So, in short limiting the SN resolution is very beneficial in those cases when operating over constrained networks at low throughput.
    But this is only one piece of the story. Configuring the SN in this way allows to limit the wire overhead to 2 bytes. So, where are the other 2 bytes?
    Luca Cominardi
    @Mallets
    This is in the data message itself and how resources are represented:
    • In zenoh each resource is represented by a key, in the form of URI. E.g. /home/sensor/temperature
    • The key can be arbitrary long. This is very handy for the user who is completely free to define his own key structure, as complex/nested as he wants to be.
    • However, transmitting the whole key is not wire efficient at all. For this reason, in zenoh we have a mechanism to do some dynamic mapping between the complete string form of the key into a more compact integer form, called simply resource ID. This means that we have a wire-efficent mapping for identifying resources departing from simple integers sent on the wire. This resource ID is encoded as VLE, thus the possibility to use only 1 byte.
    • In order to use this wire-efficient representation you may need to use the declare_resource() primitive in zenoh, which does exactly that under the hood.
    So, summarizing. 4 bytes are the minimal overhead that can be achieved by zenoh thanks to:
    1) SN negotiation and limited to 1 byte. This might have an impact on the achievable throughput due to max num of messages that can be on the fly at the same time.
    2) The efficient mapping between string and integer resources on the wire.
    @OliLay I hope this clarify your doubt
    halfbit
    @halfbit:matrix.org
    [m]
    that's really really nice
    halfbit
    @halfbit:matrix.org
    [m]
    with that sync branch, I get 1.5 million msgs/sec with the shm thr samples, amazing
    so... yeah I think removing some async stuff there really did do some magic
    Luca Cominardi
    @Mallets
    Yes, that’s what we discovered by investigating in depth the async stack. It’s very powerful, but has a huge cost when used everywhere. So, the sync branch is really an ongoing effort of fencing the async part in few specific parts in zenoh.
    halfbit
    @halfbit:matrix.org
    [m]
    makes a lot of sense
    Oliver Layer
    @OliLay
    Hi @Mallets. Thank you very much for the information, which makes the protocol much clearer for me now. I was a bit confused, because I have zenoh TCP traffic captured, and as I saw in the source code that requires the frame length and the payload length to be prepended each. With leaving that aside (e.g. using UDP) and using a 1 byte SN as you said, I get how you can have a 4 byte data message on the wire. Thanks again :)
    kydos
    @kydos

    with that sync branch, I get 1.5 million msgs/sec with the shm thr samples, amazing

    Using shared memory is useful for payload that are at least 1Kbytes. Otherwise try w/o shared memory for smaller data… You should get even higher throughput.

    halfbit
    @halfbit:matrix.org
    [m]
    kydos (Angelo Corsaro) for my message size avg of about 512 bytes, shm does about 100-200k msgs/s more
    you know, on my desktop, not necessarily a great comparison given a lot of these things are likely to be small embedded linux arm devices, still impressive to me
    kydos
    @kydos
    Cool @halfbit:matrix.org, glad to hear that you are happy about the performance level.
    Luca Cominardi
    @Mallets
    @OliLay yes, right. In case of operating over TCP we need to add 2 bytes to identify the actual batch length on the wire. This is not needed on UDP or e.g. QUIC. Moreover, to complete really the story, the 2 bytes for TCP + 2 bytes for SN number are sent only once per wire-frame. Each frame then can contain multiple data messages. E.g., considering a frame batch is max 64KB, you can fit in it thousands of 8 bytes tiny messages. For each of those message you get the minimal 2 bytes overhead for data messages. But then, the 2 bytes of TCP and 2 bytes of the SN are sent only once for all those thousands of tiny messages.
    Oliver Layer
    @OliLay
    @Mallets yes, I get that. Thanks :)
    halfbit
    @halfbit:matrix.org
    [m]
    whats a <LOCATOR> in the args list:
    Julien Enoch
    @JEnoch
    That’s the « address » of a transport endpoint in the format "<transport>/<address>". e.g.:
    • tcp/123.4.5.6:7447 (or tcp/eu.zenoh.io:7447)
    • tls/123.4.5.6:7447
    • udp/123.4.5.6:7447
    • quic/123.4.5.6:7447
      ...
    halfbit
    @halfbit:matrix.org
    [m]
    also is nats-bench somehow cheating? seems like a peer to peer connection would be significantly faster than one with a broker involved
    nats-bench is just a hair behind this thing
    which... is surprising, given I imagine at least a few additional copies and syscalls are involved
    Luca Cominardi
    @Mallets
    A locator is the ‘place’ where you make yourself available and others can connect to: in TCP/UDP terms it’s IP:PORT.
    Examples are locators are:
    • tcp/192.168.1.1:7447
    • tcp/mydoing.com:7447
    • tcp/localhost:7447
    • unixsock-stream//tmp/mysock.sock
    Ok, I see that @JEnoch replied before me :D
    Tbh we didn’t have a look into nats-bench deeply, so I don’t really know about the inner details… what do you experience?
    halfbit
    @halfbit:matrix.org
    [m]
    ah ok that makes sense, I figured it'd just take the ip addr but it didn't like that
    nats-bench pushing 512 byte messages to a local nats server, with a single publisher/subscriber gets me 1.4 million msgs/s
    I'd rather see latency histograms than throughput though to be honest
    kydos
    @kydos
    @halfbit:matrix.org, what are you using for benchmarking NATS, dds-bench? If so this is not really apple to apple comparison, please take a look at this line https://github.com/nats-io/nats.go/blob/3b1f6fcc1e1014c838036367494c3012523166b0/examples/nats-bench/main.go#L168
    halfbit
    @halfbit:matrix.org
    [m]
    nats-bench
    they have their own bench tool which I mean, yeah, hard to say its doing even remotely the same stuff
    kydos
    @kydos
    Their test is written in such a way to do user-level batching/flushing — real-world application rarely do that manually.
    halfbit
    @halfbit:matrix.org
    [m]
    yeah we don't do that in using nats
    kydos
    @kydos
    In zenoh if you look at the code for pub/sub used for throughput that’s what a beginner would write. We think that is important as your application will not need to get into these nasty details to get decent performances
    Our scheduling is done at the transport level, thus it is transparent for the user.
    halfbit
    @halfbit:matrix.org
    [m]
    its interesting that batching is even needed, I get the ideal scenario is avoiding too many sys calls... would io_uring do better without having to batch up messages?
    push messages into a queue with the kernel, let it do things as fast as it can kind of deal