On a native PC, I launch:./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.89.76.138:7447 -e tcp/10.89.76.139:7447
On a container running on another PC, I launch:./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.92.70.153:7447 -e tcp/10.89.76.138:7447
The container's host is forwarding port tcp/7447 into the container. I confirmed that port forwarding is working with iperf
.
However when I run a ROS 2 publisher on the native PC, and then list topics in the container, I don't see the published topic.
zenoh-bridge-dds
talking to each other in wireshark
zenoh-bridge-dds
keeps making the topic available even though there are no publishers for it anymore.
Hi @geoff_or:matrix.org,
On a native PC, I launch:
./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.89.76.138:7447 -e tcp/10.89.76.139:7447
On a container running on another PC, I launch:./target/release/zenoh-bridge-dds -m peer -d 0 -l tcp/10.92.70.153:7447 -e tcp/10.89.76.138:7447
A single TCP connection is good enough for 2 zenoh peer to communicate with each other (as it's a bi-directional link). Usually, a container only accepts incoming connections and cannot establish outgoing connection. Thus, you can remove the -e tcp/10.89.76.138:7447
argument from the bridge on container (but it dosn't harm, there will be no duplicate traffic).
However when I run a ROS 2 publisher on the native PC, and then list topics in the container, I don't see the published topic.
For scalability purpose, the zenoh-dds-bridge doesn't route all the DDS discovery traffic (if a DDS system declares thousands of Readers/Writers, we don't want the DDS discovery traffic to overwhelm all the other bridged systems that might need only few topics). Only user's publications are routed, if there is a matching subscription somewhere.
In detail, here is how 1 bridge works:
Notice that I'm currently working on an admin space for the zenoh-bridge-dds (similar to the zenoh router's admin space). With this, you'll be able to browse the discovered DDS Reader/Writer and established route for each zenoh-bridge-dds. A zenoh admin space being accessible via GET operations using any zenoh API (including the REST API).
Interestingly, in captured packet 19 I can see the IP address of the container (10.92.70.153) being sent to the native PC (10.89.76.138). I don't know how zenoh discovery works but is it possible that the instance of zenoh running inside the container is telling the instance running on the native PC to connect to the container's IP rather than the IP of the host PC (which would then NAT and forward that connection into the container)?
That's to be confirmed ( @OlivierHecart ?), but I think this packet 19 is part of the "gossip discovery": each zenoh peer send to other peers the list of locators it discovered. Thus another peer will try to connect those locators to establish a more robust graph of connectivity. Again, if ever this connection is established in parallel of an already existing connection, it doesn't harm.
I thikn that's also why communication worked across the router. Topic discovery doesn't seem to, but if I publish the same topic locally then ROS thinks the topic exists now and will start publishing it
Yep, that's what's happening. I have to publish a topic in the container to make ROS think it exists before I can echo it. So topic discovery across the DDS bridge isn't working.
I don't understand why you needs to "publish a topic in the container to make ROS think it exist". Does ROS creates the DDS subscription only if it discovers the DDS publication ? That would explain the behaviour you see.
But that's not what I saw when testing with the ROS2 turtlesim + teleop: both are declaring their DDS subsciptions at startup anyway.
It's working using a router on the host PC. I can communicate over ROS between the native PC and the container :)
I'd like to get it going over NAT without the router though, if that's possible.
It should have been working with your 1st configuration, without the router.
Maybe you have been confused by the ros2 topic list
not working because of non-routed discovery traffic ?
Can you please give it a try again, starting the ROS2 nodes on each side and cheking if the data flows ? You can also set the RUST_LOG=debug
environment variable to see the DDS discovery information and the zenoh established routes.
I don't understand why you needs to "publish a topic in the container to make ROS think it exist". Does ROS creates the DDS subscription only if it discovers the DDS publication ? That would explain the behaviour you see.
But that's not what I saw when testing with the ROS2 turtlesim + teleop: both are declaring their DDS subsciptions at startup anyway.
The case you were testing will work the way you describe, but that's effectively for a finished system. It works because the nodes you run on both sides of the bridge explicitly open up the topics they want. But there are a lot of things in ROS that don't work that way. For example, no ROS developer is going to be happy if the developer tools don't work, and the developer tools won't work if introspection information isn't available. (There are important use cases in ROS where having that information available without having to have a shell on the actual robot is important.) That's why ros2 topic list
won't show topics that are actually available - it doesn't subscribe to them to list them. It's also why ros2 topic echo
won't echo a topic that actually exists on the far side of the bridge: it thinks the topic doesn't exist because it can't see it in the discovered topics and so it refuses to subscribe to it. And it's why rviz will not be able to visualise data by topic - it doesn't know the topics exist, so the developer must know in advance.
ros2 topic *
commands that relies on DDS discovery ?ros2 topic
command is implemented here:ros2 topic list
won't work, I've confirmed that running the zenoh-dds-bridge
in peer mode across NAT without a router on the host PC works just fine. Thanks!
RUST_LOG=debug
environment variable prior to start it), do you see in logs a route being created for the topic you’r subscribing to ?New route: DDS ‘…’ => zenoh ‘…’
DiscoveredPublication
and the name of your topic.I have found the log message,
firstly I start the zenoh bridge dds and it gives me the 'New Router DDS ' message.
Then I started the ros2 node hello_publisher with the topic name hello_world.
The log message shows:
[2021-05-17T09:52:36Z DEBUG zenoh_bridge_dds] DiscoveredPublication(rt/hello_world, std_msgs::msg::dds_::String_, None
[2021-05-17T09:52:36Z DEBUG zenoh_bridge_dds] Declaring resource /rt/hello_world
[2021-05-17T09:52:36Z DEBUG zenoh::net::routing::resource] Register resource /rt/hello_world
[2021-05-17T09:52:36Z INFO zenoh_bridge_dds] New route: DDS 'rt/hello_world' => zenoh '/rt/hello_world' (rid=16) with type 'std_msgs::msg::dds_::String_'
[2021-05-17T09:52:36Z DEBUG zplugin_dds] Local Domain Participant IH = 10456347892458033829
[2021-05-17T09:52:36Z DEBUG zplugin_dds] Discovery data from Participant with IH = 10456347892458033829
[2021-05-17T09:52:36Z DEBUG zplugin_dds] Discovered endpoint is keyless: true
[2021-05-17T09:52:36Z DEBUG zplugin_dds] Ignoring discovery from local participant: rt/hello_world
After that I started on my local machine the python script, the log message showed :New session link established from 6F2AC0C5A89D4050AC9473305CD6E25C: tcp/xx.xx.9.50:7447 => tcp/xx.xx.8.224:33470
I think the connection is already built, and the zenoh bridge has also found the publisher on topic hello_world.
[2021-05-17T10:05:43Z DEBUG zenoh::net::routing::resource] Register resource rt/hello_world
[2021-05-17T10:05:43Z DEBUG zenoh::net::routing::pubsub] Register peer subscription rt/hello_world (peer: 6F2AC0C5A89D4050AC9473305CD6E25C)