by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Robin Luk
    @lubinsz
    Maybe you can try to use tcpdump to capture the veth interface on your linux bridge
    Find out where the packet was dropped
    ooyyloo
    @ooyyloo
    I notice that there's no ip address of eth0 when I run ifconfig. Is there something wrong with the IP address allocation of docker container?
    Ian Lewis
    @ianlewis
    You mean if you run ifconfig on the host?
    gVisor takes the IP address for the device in the network namespace so that the host doesn't interfere with networking. For example, gVisor's network stack in entirely in userspace and some traffic stays entirely within the sandbox without hitting the host. It would be inconsistent and counter intuitive if you could apply routing or config to that address that wasn't applied because the traffic was never handled by the host.
    granted, it does seem a bit weird at first
    ooyyloo
    @ooyyloo

    You mean if you run ifconfig on the host?

    No. I run ifconfig in the runsc container(I use nsenter to enter the runsc container). No IP address displayed.
    I think as there's no IP allocated for eth0, it will not send ARP response. But the container itself has its own IP address, and I have used the address to send ARP query packets to the container, and the container has received the ARP query packets.
    You can see more specific info at google/gvisor#2835.

    eth0      Link encap:Ethernet  HWaddr 02:42:ac:xx:xx:xx
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:33 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:1722 (1.7 KB)  TX bytes:0 (0.0 B)
    Ian Lewis
    @ianlewis
    You use nsenter on the host right? You aren't execing into the sandbox?
    I guess what I'm saying is that it's not abnormal to for the IP address to not be set on the device in the netns after the sandbox has started
    i.e. if you run ip netns exec <netns> ip addr from the host
    ooyyloo
    @ooyyloo
    How should I solve this networking problem? I am not familiar with the networking of gVisor. I don't know what I can do now.
    Bhasker Hariharan
    @hbhasker
    @ooyyloo What is the exact problem you are facing?
    gVisor steals the address from the veth device and assigns it to its internal NIC. We need to do that because if the IP is still assigned to the veth device the host kernel will respond to the incoming packet and in case of TCP for example it will send RSTs because the connections managed by gVisor user-space stack will not be visible to the kernel.
    ooyyloo
    @ooyyloo
    My problem is, the container which is running on runc cannot connect to the container running on runsc. "cannot connect" here means "ping" fails and "http request" fails. (I have a service running in the runsc container, and When I issue a http request to the runsc container, an "EHOSTUNREACH" error occurs.)
    ooyyloo
    @ooyyloo
    I use "docker-compose up --scale runner=2 --build" to start containers.
    And the docker-compose.yml:
      runner:
        image: ...
        runtime: runsc
        restart: always
    
      haproxy:
        image: dockercloud/haproxy
        links:
          - runner
        ports:
          - 8221:8221
        restart: unless-stopped
    
      server:
        runtime: runc
        links:
          - haproxy
        ...
    Bhasker Hariharan
    @hbhasker
    hmm I am not that familiar with docker-compose but could you put some details on what your network setup looks like
    how are the two containers connected and what routes are setup
    ooyyloo
    @ooyyloo

    How are the two containers connected

    docker-compose will create a default network whose mode is 'bridge' to make the two containers connected.

    What routes are setup

    Below, I use 'route' command to print routing table.

    host

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    default         172.27.0.1      0.0.0.0         UG    0      0        0 eth0
    172.17.0.0      *               255.255.0.0     U     0      0        0 docker0
    172.18.0.0      *               255.255.0.0     U     0      0        0 br-19946452c8f1
    172.27.0.0      *               255.255.240.0   U     0      0        0 eth0

    runsc container ("runner")

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

    (The routing table of this container is empty)

    The runc container ("server")

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    default         172.18.0.1      0.0.0.0         UG    0      0        0 eth0
    172.18.0.0      *               255.255.0.0     U     0      0        0 eth0
    Bhasker Hariharan
    @hbhasker
    Thanks let me setup something similar and see what might be going on
    Ridwan Sharif
    @ridwanmsharif
    :wave: How do I start an ubuntu docker container (using the runsc runtime) and run bash and be able to run apt-get update. Do I need to run with some special runtime args? Like --network=host? It seems to fail cause of NO_PUBKEY issues and when using the --network=host flag, apt-get update just exits the container
    Bhasker Hariharan
    @hbhasker
    does docker run --runtime=runsc ubuntu -it /bin/bash not work?
    err docker run -it --runtime=runsc ubuntu
    Jianfeng Tan
    @tanjianfeng
    Hi, a quick question, why is secio (pkg/secio/) necessary in sentry?
    Jianfeng Tan
    @tanjianfeng
    Seems just to provide the io.Reader/Writer
    Jianfeng Tan
    @tanjianfeng
    In commit 6fa5cee82c0f ("Prevent memory leaks in ilist"), Tamir Duberstein fixes a bug in ilist. Just wonder if there's a real case triggering this bug?
    Bhasker Hariharan
    @hbhasker
    I believe we fixed it when we noticed a big in netstack but that wasn't the root cause of the bug just something we observed. That an item removed from the list could still cause a chain of list to be retained if something was holding on to the item that was just removed
    Because it's prev/next pointers still pointed to elements on the list
    Jianfeng Tan
    @tanjianfeng
    @hbhasker Do you think the problem you mentioned is specific to netstack? We use another network stack, but encouter a similar issue. With heap profile, a lot of objects leak pointed by RecordSocket.
    Bhasker Hariharan
    @hbhasker
    Are you using the gvisor list?
    There are a few leak patterns that can happen. List was just one. Slicing an array that contains pointers will leak as well
    We had bugs like that
    Jianfeng Tan
    @tanjianfeng
    k.sockets (socketList) is indeed based on pkg/ilist. We can understand it would affect anything based pkg/ilist. We are trying to find out how to reproduce it. Anyway, we shall backport this fix ASAP.
    Bhasker Hariharan
    @hbhasker
    ah yea then it could leak if for example a socket was removed from the list but a reference was held to it then it can cause a lot of other closed sockets to not be GCed as the prev/next pointers will keep them alive
    Jianfeng Tan
    @tanjianfeng
    Many thanks! @hbhasker
    Bhasker Hariharan
    @hbhasker
    @tanjianfeng did the fix work?
    Robin Luk
    @lubinsz
    Hey,
    Does gVIsor has the plan to support illumos & illumos-kvm?
    Robin Luk
    @lubinsz
    When I participated in the k8s weekly meeting, I saw that sig-arch was discussing whether to support illumos, so I wonder if gVisor also has this plan
    Adin Scannell
    @amscanne
    Interesting, I didn’t realize they had ported kvm directly
    We have never discussed Illuminos.. to the best of my knowledge, this is the first time
    Robin Luk
    @lubinsz
    I see.
    Thanks.
    Adin Scannell
    @amscanne
    Reading about their kvm port however, it seems like they don’t have mmu notifiers?
    Given that it forked from 2.6.34, I suspect there would be a number of gaps for KVM, but maybe not insurmountable
    (I would be concerned about the lack of mmu notifiers and what that means for the way the platform works fundamentally... it means that at least the sentry binary would be locked, as would any other host page cache pages used)
    But no immediate plans to support Illuminos in any case
    Jianfeng Tan
    @tanjianfeng

    @tanjianfeng did the fix work?

    @hbhasker We are testing the new version. It'll be online later in this week. Will report back the result. Thanks!

    Muhammad Yuga Nugraha
    @myugan
    HI
    Robin Luk
    @lubinsz
    Hi
    gattytto
    @gattytto
    hi