Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Angelo Corsaro
    @kydos

    The zenoh project was started to address the short comings of existing technologies in addressing hetherogeneity, scale and integration between data at rest and data in motion. In other terms we wanted to have a protocol with the following characteristics:

    • Extremely wire efficient: to be used effectively on low power networks

    • Low static and dynamic footprint: to be able to run on anything ranging from a server to a micro-controller

    • Scalable to Internet-scale applications

    • Unified abstraction for data at rest and data in motion: a minimal set of abstractions to dealing with data in motion as well as at rest

    As a protocol zenoh is an NDN as its addressing is based on naming data. Additionally the query protocol used in zenoh is heavily influenced by NDN research. On the other hand, and in contrast with traditional NDN protocols, zenoh also support extremely efficient pub/sub. Thus in a very superficial way, you could say that zenoh tried to merge the best of NDN- and DDS-alike and added some of its own sauce to the mix.
    Angelo Corsaro
    @kydos
    If we try to compare and contrast DDS to zenoh, there are quite a few differences. One of the most important at a wire-protocol level concerns how reliability is implemented. In DDS there is a reliable run between each publisher and matching subscriber. As a consequence the dynamic footprint of a publisher depends (linearly) on the number of matching subscribers. Zenoh reliability is between protocol runtimes and does not depends on the number of matching readers/writers. This as a series of consequences on discovery overhead as well as on the amount of state that has to be retained on nodes. The net consequence is that zenoh applications do not have a strict dependency between runtime footprint and the scale of your system. This is an essential for scaling out.
    Finally, both zenoh as well as CycloneDDS are project ran by the ADLINK Advanced Technology Office (which I happen to run).
    zenoh is the protocol that we use to route and internconnect across a WAN CycloneDDS applications, as well as the protocol that we use to provide integrated CycloneDDS applications with extremely constrained devices.
    A zenoh-router plugin for DDS will be available in a matter of a few weeks.
    @Harmouch101, let me know if you have any further questions.
    Angelo Corsaro
    @kydos
    @methylDragon I’ll take a look at the FFI problem and let you know my findings ASAP. Thanks for the extremely well detailed github issue.
    methylDragon
    @methylDragon
    No problem!
    OlivierHecart
    @OlivierHecart

    Hello @carelphillipkruger , in the current rust implementation of zenoh, routers do not try to automatically connect to each other. The main reason of that is that we want the user to keep strong control on the routing network topology (we can discuss that further if needed). So you need to explicitly connect routers with each other and set up the exact topology you want.
    The -p option is indeed gone and has been replaced by the -e option. So to interconnect two routers running on a same host :

    zenohd -l tcp/0.0.0.0:7447
    zenohd -l tcp/0.0.0.0:7448 -e tcp/127.0.0.1:7447

    Current rust routers can only route if interconnected in a "clique" (all routers directly connected to each other). Full routing capacities is one of our next items in the roadmap. It should added in the next weeks/months.

    OlivierHecart
    @OlivierHecart

    @methylDragon , about this :

    Would it be feasible to implement a way to obtain the resource ID that a message was obtained from within a ZNSubscriber's callback function (from the ffi)?

    e.g.
    If I zn_write_wrid() to a resource ID (say, 1), could I obtain that number somehow from a ZNSubscriber's callback function (perhaps exposed in the zn_sample object that gets passed?)

    We decided not to expose this information. The main reason for that is that the resource name to resource id mapping is purely local. So, in routed communications, the received id from the network is probably different from the one that has been used by the publisher. The only case in which you would be able to get the same id as used by the publisher would be in peer2peer communications.
    May I ask why you would like to get this information so that we could reconsider our choice or provide another way to address the use case?

    Harmouch101
    @Harmouch101
    @kydos Thank you, man, for the valuable information, but I have a fundamental question that concerns the architecture of NDN that has 2 types of packets: interest and data packets, And the 3 routing tables(CS, PIT, and the ordinary routing table FIB). So how zenoh-net implements all of this stuff, or maybe it is a future work or something like that.
    Screenshot from 2020-08-24 14-57-36.png
    I mean this NDN architecture.
    Harmouch101
    @Harmouch101
    What I have understood from NDN is that in the future I could be able to access a web page/server using the following schema: com/google/seach_query instead of the ordinary URL www.google.com/seach_query and the hosts are no longer having IP addresses, instead, they will have names. I knew that zenoh implement this using the Naming Data property that is already implemented. But how about the hostnames. I think this should be implemented in the zenoh-router.
    Thanks again for your detailed responses.
    Angelo Corsaro
    @kydos
    @Harmouch101 in short, without getting into the details of the protocols, but just giving you the essential. Zenoh routing implementation is a bit more articulated than NDN. But to keep things simple, and with reference to the Rust implementation, the current version of zenoh has bpth the FIB as well as the PIT tables. We call them differently, but the information is roughly the same. The Content store is used to cache data and shorten the potential resend-path. This is not in place yet, and will be the next step.
    Concerning your example on indexing and retrieving content with NDN protocols, this is exactly what you can do with zenoh. Additionally, once we’ll have a file-system backend (see https://github.com/eclipse-zenoh/zenoh/wiki/Roadmap) you will be able to use zenoh as a geo-distributed web-server and content caching network.
    Angelo Corsaro
    @kydos
    Finally, for what concerns the host-name, that is a “configuration” detail necessary to set-up your zenoh network. Yet, this is an orthogonal aspect to how you address data. In other terms, once you have a zenoh infrastructure up an running you do not care anymore of “hosts”. You only care and address data.
    Harmouch101
    @Harmouch101
    Alright, that's epic. thank you once again for the valuable information.
    methylDragon
    @methylDragon

    @OlivierHecart thanks for answering my query! I originally wanted it to debug the issue I posted above, I think that's not necessary now, so it's all good (:

    Although, I'd like to ask if there are any suggestions on getting messages timestamped in the FFI (that is, the time the messages are sent out and received.) Would there be plans to expose that information in Zenoh, or is it preferable that I manually attach it to the message bytes I send instead?

    OlivierHecart
    @OlivierHecart
    @methylDragon , all messages can be timestamped automatically by enabling the "add_timestamp" feature at compile time (this feature will probably be activated by default in the future). The timestamp is available in the data_info in the rust API. But the data_info is not yet available through the ffi binding. This will be added too. This is for the send timestamp. We don't plan to add the receive timestamp.
    methylDragon
    @methylDragon
    ah, very nice!
    Harmouch101
    @Harmouch101
    Hello everyone,
    I hope you all doing well. So while I was researching on the pub/sub architecture and message queueing algorithms, I had the following questions that concern on the implementation of zenoh for this architecture :
    1. What happens to the sending message if it fails or the receiving subscriber is temporarily unavailable?
    2. How long are messages kept in the storage/queue?
    3. What happens if the subscriber is overloaded with messages?
      So, I wonder how zenoh can handle these situations.
      Thanks in advance, and have a great day.
    OlivierHecart
    @OlivierHecart

    Hello @Harmouch101 , what we currently have is what I would call hop-to-hop no dropping reliability implementation. Meaning that a reliable protocol (relying on TCP reliability for now) is run between each hop in the routing path. I'm not sure about the size of the internal queue but @Mallets can answer that in details. Nothing is dropped until the receiver finally acknowledges the message or is declared dead. So to answer points 1 and 3, back pressure is propagated back from all subscribers down to the publisher that may eventually been slowed down.

    We plan to add a dropping strategy to avoid slow subscribers to slow down the full system (basically older messages would be dropped when the queue is full).

    This would grant full reliability while the routing network topology is stable but there may be losses on topology changes (router crash, etc ...). So we also plan to add an end-to-end reliability mechanism by tagging each message with a writerId + writerSN which would allow subscribers to detect missed samples and ask for resends (samples would be retained by storages disseminated in the network for some period of time). This strategy provides a greater reliability but brings scalability issues.

    Luca Cominardi
    @Mallets
    Hi @Harmouch101 , regarding the size of the internal queue, this is bounded in size rather than in number of messages. The current default value the queue is set to 256KB. So, the actual number of messages that can fit in a queue depends on the size of the messages.
    Harmouch101
    @Harmouch101
    Alright @OlivierHecart, thanks. So zenoh doesn't implement the retransmission mechanism at the current moment. And according to @Mallets, the maximum size of the message that can be sent is 256 KB, so the MTU = 256 KB, right? and if it is larger than that, it will be fragmented as 256 KB segments?
    Luca Cominardi
    @Mallets
    @Harmouch101 , no. The MTU is virtually unlimited in Zenoh. This is achieved by fragmenting the data messages. Zenoh fragments the data messages little by little in chunks of 65KB max of size (the default value is 16KB). These chunks are then inserted into the transmission queue, which is indeed limited to 256KB (it can be configured).
    Harmouch101
    @Harmouch101
    Ah ok. Now I get it, thanks.
    Luca Cominardi
    @Mallets
    In other terms, we use a default MTU of 16KB and anything larger than that is fragmented. The fragments are then inserted in the transmission queue which is limited to 256KB.
    In this way we can apply backpressure. Please note that fragmentation is done hop-by-hop
    Harmouch101
    @Harmouch101
    Alright!
    Angelo Corsaro
    @kydos
    @Harmouch101, are you writing an essay/report on Pub/Sub technologies. If so do not hesitate to share with us if you want comments and or feedback.
    Harmouch101
    @Harmouch101
    Well, I am currently working on my report, my final year project, which will explain pretty much everything that zenoh uses in terms of technologies, protocols, network architecture in order to accomplish its promises. I think right now I am not allowed to share anything until I submit it at the administration of my university. In the near future, I will share it with you guys anyway.
    Harmouch101
    @Harmouch101
    If someone has the time, please take a look on my experiment that I have done hereon zenoh in term of network performance. does this seem correct? If you have any feedback on this i would appretiate that so much.
    Angelo Corsaro
    @kydos
    @Harmouch101, I’ll take a look and will share my comments.
    Angelo Corsaro
    @kydos
    Hello everyone, for those of you using the C API exposed through Rust FFI, we have now made available a function to init the runtime logger. This way you can use environment variables when starting C/C++ applications to control zenoh logs. The macro are exactly the same used in Rust (see here) while the C function to init the logger, namely void zn_init_logger(void);
    Joe Speed
    @joespeed
    Seafloor asked about using it for autonomous boat swarms via LoRa https://www.seafloorsystems.com
    Angelo Corsaro
    @kydos

    Seafloor asked about using it for autonomous boat swarms via LoRa https://www.seafloorsystems.com

    Cool, happy to learn more about their use case.

    Szymon Szantula
    @byq77

    Hello zenoh team! I have a couple of questions regarding using zenoh on MCUs . Your website http://zenoh.io/ has section that reads:

    Constrained Devices
    zenoh implementations can be extremely resource constrained. Our smallest implementation takes as little as 300 bytes footprint on an Atmel 8-bit microcontroller.

    However, in project's rodmap microcontroller support is listed as new feature planned for September.

    I'd like to test out zenoh on STM32 mcu that is connected using UART/serial to RPi4 running Cyclone DDS instance and I wonder if zenoh provides similar functionalities to this project. I would be grateful if you could provide me with some resources on how to use zenoh wih microcontrollers.

    Angelo Corsaro
    @kydos
    @byq77, the roadmap available here concerns the new version of zenoh. This version along with some updates to the protocol includes a Rust rewrite of the routing infrastructure. That said, for the current version of zenoh — in other term what you find on the master branch — you can use zenoh-c to run on micro-controllers. Finally the support for micro-controllers mentioned in the current roadmap is mostly about updating zenoh-c to the new version of the protocol. Let me know if you have any other questions.
    Angelo Corsaro
    @kydos
    @byq77 we’ve just added some further explanation here regarding what is available today and what is coming next for zenoh on micro-controllers
    Szymon Szantula
    @byq77
    @kydos Thanks! I'll look into it.
    Angelo Corsaro
    @kydos
    Hello everyone, we have a few openings on the zenoh team. Ideally working with us here in Paris, otherwise we can also accomodate tele-working. Please spread the word and let me know if there is any interest! Merci!
    Angelo Corsaro
    @kydos
    Hello everyone, we have an article featuring #zenoh on the latest Eclipse Newsletter: https://www.eclipse.org/community/eclipse_newsletter/2020/september/1.php
    For those of you that want to try out the DDS bridge for zenoh described in the article, check out https://github.com/eclipse-zenoh/zenoh-plugin-dds
    This is still in alpha version but quickly maturing.
    Michael Pöhnl
    @budrus
    Nice. Thanks for the newsletter hint. When it comes to ROS2, you would have a zenoh router with DDS plugin and a DDS rmw like rmw_cycloneDDS? Or do you also think of an rmw_zenoh?
    Angelo Corsaro
    @kydos
    Hello @budrus, we have the zenoh DDS plugin and the ROS2 team has developed an experimental RMW layer for zenoh. On our side we are working on an zenoh-pico RMW for MicroROS2. This should be available sometimes in October. We are also planning to make a zenoh integration with IceOryx. That should be available probably in mid November.