Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    halfbit
    @halfbit:matrix.org
    [m]
    at least on this puny gigabit network I have
    no 10g to test against :-)
    Geoff
    @geoff_or:matrix.org
    [m]
    I can't get the zenoh-bridge-dds to work across NAT ☹️

    On a native PC, I launch:
    ./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.89.76.138:7447 -e tcp/10.89.76.139:7447

    On a container running on another PC, I launch:
    ./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.92.70.153:7447 -e tcp/10.89.76.138:7447

    The container's host is forwarding port tcp/7447 into the container. I confirmed that port forwarding is working with iperf.

    However when I run a ROS 2 publisher on the native PC, and then list topics in the container, I don't see the published topic.

    Geoff
    @geoff_or:matrix.org
    [m]
    I can see the two instances of zenoh-bridge-dds talking to each other in wireshark
    Geoff
    @geoff_or:matrix.org
    [m]
    That's a wireshark capture from the host
    Interestingly, in captured packet 19 I can see the IP address of the container (10.92.70.153) being sent to the native PC (10.89.76.138). I don't know how zenoh discovery works but is it possible that the instance of zenoh running inside the container is telling the instance running on the native PC to connect to the container's IP rather than the IP of the host PC (which would then NAT and forward that connection into the container)?
    Geoff
    @geoff_or:matrix.org
    [m]
    Is there any documentation on what configuration parameters can be set in a configuration file?
    Geoff
    @geoff_or:matrix.org
    [m]
    It's working using a router on the host PC. I can communicate over ROS between the native PC and the container :)
    I'd like to get it going over NAT without the router though, if that's possible.
    Interesting... if I stop publishing the topic, it's still listed as available. I guess that zenoh-bridge-dds keeps making the topic available even though there are no publishers for it anymore.
    That's not ideal.
    I thikn that's also why communication worked across the router. Topic discovery doesn't seem to, but if I publish the same topic locally then ROS thinks the topic exists now and will start publishing it
    Yep, that's what's happening. I have to publish a topic in the container to make ROS think it exists before I can echo it. So topic discovery across the DDS bridge isn't working.
    Julien Enoch
    @JEnoch

    Hi @geoff_or:matrix.org,

    On a native PC, I launch:
    ./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.89.76.138:7447 -e tcp/10.89.76.139:7447
    On a container running on another PC, I launch:
    ./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.92.70.153:7447 -e tcp/10.89.76.138:7447

    A single TCP connection is good enough for 2 zenoh peer to communicate with each other (as it's a bi-directional link). Usually, a container only accepts incoming connections and cannot establish outgoing connection. Thus, you can remove the -e tcp/10.89.76.138:7447 argument from the bridge on container (but it dosn't harm, there will be no duplicate traffic).

    However when I run a ROS 2 publisher on the native PC, and then list topics in the container, I don't see the published topic.

    For scalability purpose, the zenoh-dds-bridge doesn't route all the DDS discovery traffic (if a DDS system declares thousands of Readers/Writers, we don't want the DDS discovery traffic to overwhelm all the other bridged systems that might need only few topics). Only user's publications are routed, if there is a matching subscription somewhere.
    In detail, here is how 1 bridge works:

    • it creates a DDS Participant that discovers all DDS Readers and Writers via regular DDSI protocol (UDP multicast by default).
    • For each discovered DDS Writer, the bridge creates a matching DDS Reader and declares a publication on a zenoh resource corresponding to the DDS topic name (and partition)
    • For each discovered DDS Reader, the bridge creates a matching DDS Writer and declares a subscription on a zenoh resource corresponding to the DDS topic name (and partition)
    • And the way zenoh works, the published DDS data routed by the bridge will be transmitted to another bridge only if this one declares a subscription.

    Notice that I'm currently working on an admin space for the zenoh-bridge-dds (similar to the zenoh router's admin space). With this, you'll be able to browse the discovered DDS Reader/Writer and established route for each zenoh-bridge-dds. A zenoh admin space being accessible via GET operations using any zenoh API (including the REST API).

    Interestingly, in captured packet 19 I can see the IP address of the container (10.92.70.153) being sent to the native PC (10.89.76.138). I don't know how zenoh discovery works but is it possible that the instance of zenoh running inside the container is telling the instance running on the native PC to connect to the container's IP rather than the IP of the host PC (which would then NAT and forward that connection into the container)?

    That's to be confirmed ( @OlivierHecart ?), but I think this packet 19 is part of the "gossip discovery": each zenoh peer send to other peers the list of locators it discovered. Thus another peer will try to connect those locators to establish a more robust graph of connectivity. Again, if ever this connection is established in parallel of an already existing connection, it doesn't harm.

    I thikn that's also why communication worked across the router. Topic discovery doesn't seem to, but if I publish the same topic locally then ROS thinks the topic exists now and will start publishing it
    Yep, that's what's happening. I have to publish a topic in the container to make ROS think it exists before I can echo it. So topic discovery across the DDS bridge isn't working.

    I don't understand why you needs to "publish a topic in the container to make ROS think it exist". Does ROS creates the DDS subscription only if it discovers the DDS publication ? That would explain the behaviour you see.
    But that's not what I saw when testing with the ROS2 turtlesim + teleop: both are declaring their DDS subsciptions at startup anyway.

    It's working using a router on the host PC. I can communicate over ROS between the native PC and the container :)
    I'd like to get it going over NAT without the router though, if that's possible.

    It should have been working with your 1st configuration, without the router.
    Maybe you have been confused by the ros2 topic list not working because of non-routed discovery traffic ?
    Can you please give it a try again, starting the ROS2 nodes on each side and cheking if the data flows ? You can also set the RUST_LOG=debug environment variable to see the DDS discovery information and the zenoh established routes.

    Geoff
    @geoff_or:matrix.org
    [m]
    Thanks for the advice. I'll give it another go in the morning.
    Geoff
    @geoff_or:matrix.org
    [m]

    I don't understand why you needs to "publish a topic in the container to make ROS think it exist". Does ROS creates the DDS subscription only if it discovers the DDS publication ? That would explain the behaviour you see.
    But that's not what I saw when testing with the ROS2 turtlesim + teleop: both are declaring their DDS subsciptions at startup anyway.

    The case you were testing will work the way you describe, but that's effectively for a finished system. It works because the nodes you run on both sides of the bridge explicitly open up the topics they want. But there are a lot of things in ROS that don't work that way. For example, no ROS developer is going to be happy if the developer tools don't work, and the developer tools won't work if introspection information isn't available. (There are important use cases in ROS where having that information available without having to have a shell on the actual robot is important.) That's why ros2 topic list won't show topics that are actually available - it doesn't subscribe to them to list them. It's also why ros2 topic echo won't echo a topic that actually exists on the far side of the bridge: it thinks the topic doesn't exist because it can't see it in the discovered topics and so it refuses to subscribe to it. And it's why rviz will not be able to visualise data by topic - it doesn't know the topics exist, so the developer must know in advance.

    Julien Enoch
    @JEnoch
    That’s exactly the piece of information we were missing about ROS2 design, thanks!
    Do you also know if those dev tools only rely on DDS Topics discovery, but also on DDS Readers/Writers discovery ?
    Better, can you point me to the implementation code of ros2 topic * commands that relies on DDS discovery ?
    So we can figure out if there is something smart we could do, without bringing back a overwhelming DDS discovery traffic.
    kydos
    @kydos
    @JEnoch and @geoff_or:matrix.org I think that we should provide an alternative implementation for ros2 topic based on zenoh queries.
    2 replies
    Julien Enoch
    @JEnoch
    @kydos according to Geoff, I understood that would not solve the issues for rviz.
    Gergely Kis
    @kisg
    I have not yet tried it myself, but Iceoryx does have a DDS bridge using CycloneDDS: https://github.com/eclipse-iceoryx/iceoryx/tree/master/iceoryx_dds
    I don't know if their implementation was done with the above requirements in mind, but it might be worth a look.
    Geoff
    @geoff_or:matrix.org
    [m]
    @JEnoch: The ros2 topic command is implemented here:
    https://github.com/ros2/ros2cli/tree/master/ros2topic/ros2topic
    Geoff
    @geoff_or:matrix.org
    [m]
    @JEnoch: Knowing now that ros2 topic list won't work, I've confirmed that running the zenoh-dds-bridge in peer mode across NAT without a router on the host PC works just fine. Thanks!
    halfbit
    @halfbit:matrix.org
    [m]
    in testing iceoryx today, that I did a thorough job, it didn't really seem to be much faster than zenoh for my small message scenario, the bonus of zenoh being I don't need to suddenly jump to another protocol for multi-host pub/sub
    *not that I did a thorough job
    yingliangzhe
    @yingliangzhe:matrix.org
    [m]
    Hello Everyone,
    I am trying to subscribe a message from ros2. zenoh dds bridge is already opened on the remote machine. The ros2 is running on remote machine, and my python script is running on local machine. I have sucessfully published a message from my local machine and the remote one has already received it. But If I tried to receive messages from the remote machine, it seems that, the callback function is not called.
    Does someone has the same experience?
    Julien Enoch
    @JEnoch
    Hi @yingliangzhe:matrix.org ,
    if you activate the debug logs for the bridge (exporting RUST_LOG=debug environment variable prior to start it), do you see in logs a route being created for the topic you’r subscribing to ?
    The log should be something like:
    New route: DDS ‘…’ => zenoh ‘…’
    yingliangzhe
    @yingliangzhe:matrix.org
    [m]
    for the zenoh publisher and ros2 subscriber I did see the log New route: DDS '...'
    But for the zenoh subscriber and ros2 publisher did not
    New session link established from 1BC92346AA5B4E54BB9215D8B0B4FD3F: tcp/xx.xx.9.50:7447 => tcp/xx.xx.8.224:33282
    I have only seen this log
    Julien Enoch
    @JEnoch
    Is your Python script using the zenoh API or the ROS2 API ?
    yingliangzhe
    @yingliangzhe:matrix.org
    [m]
    That uses the zenoh API. I only have the ros2 on remote machine.
    And I noticed, that the 'New route' log message will be showed on the terminal, even if I just started the ros2 node, the session from my python script has not been opened.
    I can see it, even if I only start the RUST_LOG=info
    Julien Enoch
    @JEnoch
    Yes, the bridge creates the routes as soon as it detects a ROS2/DDS publisher or subscriber. No matter if there is a remote publisher or subscriber. (those are local routes actually, between DDS protocol and zenoh protocol).
    If you don’t see a route for the DDS publication you want to subscribe to, it means that the bridge didn’t discovered this DDS publisher. In debug logs, search for a log containing DiscoveredPublication and the name of your topic.
    If you can’t find one, it means your ROS2 is not declaring this publication.
    yingliangzhe
    @yingliangzhe:matrix.org
    [m]

    I have found the log message,
    firstly I start the zenoh bridge dds and it gives me the 'New Router DDS ' message.
    Then I started the ros2 node hello_publisher with the topic name hello_world.
    The log message shows:

    [2021-05-17T09:52:36Z DEBUG zenoh_bridge_dds] DiscoveredPublication(rt/hello_world, std_msgs::msg::dds_::String_, None
    [2021-05-17T09:52:36Z DEBUG zenoh_bridge_dds] Declaring resource /rt/hello_world
    [2021-05-17T09:52:36Z DEBUG zenoh::net::routing::resource] Register resource /rt/hello_world
    [2021-05-17T09:52:36Z INFO  zenoh_bridge_dds] New route: DDS 'rt/hello_world' => zenoh '/rt/hello_world' (rid=16) with type 'std_msgs::msg::dds_::String_'
    [2021-05-17T09:52:36Z DEBUG zplugin_dds] Local Domain Participant IH = 10456347892458033829
    [2021-05-17T09:52:36Z DEBUG zplugin_dds] Discovery data from Participant with IH = 10456347892458033829
    [2021-05-17T09:52:36Z DEBUG zplugin_dds] Discovered endpoint is keyless: true
    [2021-05-17T09:52:36Z DEBUG zplugin_dds] Ignoring discovery from local participant: rt/hello_world

    After that I started on my local machine the python script, the log message showed :
    New session link established from 6F2AC0C5A89D4050AC9473305CD6E25C: tcp/xx.xx.9.50:7447 => tcp/xx.xx.8.224:33470

    I think the connection is already built, and the zenoh bridge has also found the publisher on topic hello_world.

    received this message after declaring the subscriber within a python script on local machine
    [2021-05-17T10:05:43Z DEBUG zenoh::net::routing::resource] Register resource rt/hello_world
    [2021-05-17T10:05:43Z DEBUG zenoh::net::routing::pubsub] Register peer subscription rt/hello_world (peer: 6F2AC0C5A89D4050AC9473305CD6E25C)
    Julien Enoch
    @JEnoch
    And does your Python script well subscribes to /rt/hello_world (with starting /) ?
    yingliangzhe
    @yingliangzhe:matrix.org
    [m]
    No, I did not!!! Thank you so much!!!
    That is the point
    Julien Enoch
    @JEnoch
    Great! Glad to help :smile:
    yingliangzhe
    @yingliangzhe:matrix.org
    [m]
    👍️