gvisor/community

Robin Luk

@lubinsz

Maybe you can try to use tcpdump to capture the veth interface on your linux bridge

Find out where the packet was dropped

ooyyloo

@ooyyloo

I notice that there's no ip address of eth0 when I run ifconfig. Is there something wrong with the IP address allocation of docker container?

Ian Lewis

@ianlewis

You mean if you run ifconfig on the host?

gVisor takes the IP address for the device in the network namespace so that the host doesn't interfere with networking. For example, gVisor's network stack in entirely in userspace and some traffic stays entirely within the sandbox without hitting the host. It would be inconsistent and counter intuitive if you could apply routing or config to that address that wasn't applied because the traffic was never handled by the host.

granted, it does seem a bit weird at first

ooyyloo

@ooyyloo

You mean if you run ifconfig on the host?

No. I run ifconfig in the runsc container(I use nsenter to enter the runsc container). No IP address displayed.
I think as there's no IP allocated for eth0, it will not send ARP response. But the container itself has its own IP address, and I have used the address to send ARP query packets to the container, and the container has received the ARP query packets.
You can see more specific info at google/gvisor#2835.

eth0      Link encap:Ethernet  HWaddr 02:42:ac:xx:xx:xx
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:33 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1722 (1.7 KB)  TX bytes:0 (0.0 B)

Ian Lewis

@ianlewis

You use nsenter on the host right? You aren't execing into the sandbox?

I guess what I'm saying is that it's not abnormal to for the IP address to not be set on the device in the netns after the sandbox has started

i.e. if you run ip netns exec <netns> ip addr from the host

ooyyloo

@ooyyloo

How should I solve this networking problem? I am not familiar with the networking of gVisor. I don't know what I can do now.

Bhasker Hariharan

@hbhasker

@ooyyloo What is the exact problem you are facing?

gVisor steals the address from the veth device and assigns it to its internal NIC. We need to do that because if the IP is still assigned to the veth device the host kernel will respond to the incoming packet and in case of TCP for example it will send RSTs because the connections managed by gVisor user-space stack will not be visible to the kernel.

ooyyloo

@ooyyloo

My problem is, the container which is running on runc cannot connect to the container running on runsc. "cannot connect" here means "ping" fails and "http request" fails. (I have a service running in the runsc container, and When I issue a http request to the runsc container, an "EHOSTUNREACH" error occurs.)

ooyyloo

@ooyyloo

I use "docker-compose up --scale runner=2 --build" to start containers.
And the docker-compose.yml:

  runner:
    image: ...
    runtime: runsc
    restart: always

  haproxy:
    image: dockercloud/haproxy
    links:
      - runner
    ports:
      - 8221:8221
    restart: unless-stopped

  server:
    runtime: runc
    links:
      - haproxy
    ...

Bhasker Hariharan

@hbhasker

hmm I am not that familiar with docker-compose but could you put some details on what your network setup looks like

how are the two containers connected and what routes are setup

ooyyloo

@ooyyloo

How are the two containers connected

docker-compose will create a default network whose mode is 'bridge' to make the two containers connected.

What routes are setup

Below, I use 'route' command to print routing table.

host

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.27.0.1      0.0.0.0         UG    0      0        0 eth0
172.17.0.0      *               255.255.0.0     U     0      0        0 docker0
172.18.0.0      *               255.255.0.0     U     0      0        0 br-19946452c8f1
172.27.0.0      *               255.255.240.0   U     0      0        0 eth0

runsc container ("runner")

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

(The routing table of this container is empty)

The runc container ("server")

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.18.0.1      0.0.0.0         UG    0      0        0 eth0
172.18.0.0      *               255.255.0.0     U     0      0        0 eth0

Bhasker Hariharan

@hbhasker

Thanks let me setup something similar and see what might be going on

Ridwan Sharif

@ridwanmsharif

:wave: How do I start an ubuntu docker container (using the runsc runtime) and run bash and be able to run apt-get update. Do I need to run with some special runtime args? Like --network=host? It seems to fail cause of NO_PUBKEY issues and when using the --network=host flag, apt-get update just exits the container

Bhasker Hariharan

@hbhasker

does docker run --runtime=runsc ubuntu -it /bin/bash not work?

err docker run -it --runtime=runsc ubuntu

Jianfeng Tan

@tanjianfeng

Hi, a quick question, why is secio (pkg/secio/) necessary in sentry?

Jianfeng Tan

@tanjianfeng

Seems just to provide the io.Reader/Writer

Jianfeng Tan

@tanjianfeng

In commit 6fa5cee82c0f ("Prevent memory leaks in ilist"), Tamir Duberstein fixes a bug in ilist. Just wonder if there's a real case triggering this bug?

Bhasker Hariharan

@hbhasker

I believe we fixed it when we noticed a big in netstack but that wasn't the root cause of the bug just something we observed. That an item removed from the list could still cause a chain of list to be retained if something was holding on to the item that was just removed

Because it's prev/next pointers still pointed to elements on the list

Jianfeng Tan

@tanjianfeng

@hbhasker Do you think the problem you mentioned is specific to netstack? We use another network stack, but encouter a similar issue. With heap profile, a lot of objects leak pointed by RecordSocket.

Bhasker Hariharan

@hbhasker

Are you using the gvisor list?

There are a few leak patterns that can happen. List was just one. Slicing an array that contains pointers will leak as well

We had bugs like that

Jianfeng Tan

@tanjianfeng

k.sockets (socketList) is indeed based on pkg/ilist. We can understand it would affect anything based pkg/ilist. We are trying to find out how to reproduce it. Anyway, we shall backport this fix ASAP.

Bhasker Hariharan

@hbhasker

ah yea then it could leak if for example a socket was removed from the list but a reference was held to it then it can cause a lot of other closed sockets to not be GCed as the prev/next pointers will keep them alive

Jianfeng Tan

@tanjianfeng

Many thanks! @hbhasker

Bhasker Hariharan

@hbhasker

@tanjianfeng did the fix work?

Robin Luk

@lubinsz

Hey,

Does gVIsor has the plan to support illumos & illumos-kvm?

Robin Luk

@lubinsz

When I participated in the k8s weekly meeting, I saw that sig-arch was discussing whether to support illumos, so I wonder if gVisor also has this plan

Adin Scannell

@amscanne

Interesting, I didn’t realize they had ported kvm directly

We have never discussed Illuminos.. to the best of my knowledge, this is the first time

Robin Luk

@lubinsz

I see.

Thanks.

Adin Scannell

@amscanne

Reading about their kvm port however, it seems like they don’t have mmu notifiers?

Given that it forked from 2.6.34, I suspect there would be a number of gaps for KVM, but maybe not insurmountable

(I would be concerned about the lack of mmu notifiers and what that means for the way the platform works fundamentally... it means that at least the sentry binary would be locked, as would any other host page cache pages used)

But no immediate plans to support Illuminos in any case

Jianfeng Tan

@tanjianfeng

@tanjianfeng did the fix work?

@hbhasker We are testing the new version. It'll be online later in this week. Will report back the result. Thanks!

Muhammad Yuga Nugraha

@myugan

Robin Luk

@lubinsz

gattytto

@gattytto

Where communities thrive

People

Repo info

Activity

How are the two containers connected

What routes are setup

host

runsc container ("runner")

The runc container ("server")