Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Fabricio Voznika
    @fvoznika
    We should make things like runsc list a bit more resilient in case old state files were left behind or got corrupted, but there is no guarantee that list will be able to list old containers for example.
    Marek
    @majek
    How to profile runsc or guest pids, under kvm? We got perf to do nice flame graphs for pthread, but not sure how to handle kvm.
    Does gvisor (kvm/pthread) support ASLR?
    prattmic
    @prattmic:matrix.org
    [m]
    To the first question, you have to enable guest profiling for kvm (perf record -e cycles:hg), however I don't recall if perf handles the guest symbolization well. I think you might just get a ton of "unknowns" for the guest.
    To the second question, we support ASLR for applications (actually we don't support turning it off...), but we don't have (binary) ASLR for runsc at the moment due to limitations with Go support for PIE binaries
    Marek
    @majek
    I'm doing some networking benchmarks for gvisor, maybe you'll find this interesting. I have an echo server and echo client. The echo client is sending 40 times a bulk transfer of 4MiB and waiting for it to come back
    first scenario, server on host, client in guest (gvisor)
    numbers-max.png
    the whiskers show min/max in us (per transfer of 4MiB block), the block is supposed to be avg +- deviation.
    numbers-max.png
    this is second scenario, where both tcp server and tcp echo client are inside gvisor
    Marek
    @majek
    There are two takeaways:
    • if you have a server on host and a guest in gvisor, then passing --network=host to runsc gives you 2x perf boost
      • if you have unix domain sockets inside runsc then the performance is tragic. 10x worse than anything else
    is the tragiacal performance of unix domain sockets inside runsc (both server and client, using /tmp/pipe endpoint path) known?
    am I expected to raise a bug?
    Bhasker Hariharan
    @hbhasker
    Unix domain sockets are slow becausr the cureent default buffer size is 32kb
    Could you run the test after bumping that up
    For host yes it is faster simply because linux can move skbs around. Whereas netstack has to copy bytes back and forth from the host packet socket.
    There are othe things too like our buffer management etc
    Btw i believe you will see similar crappy perfomance for udp as well since it defaults to really small buffer sizes
    Bhasker Hariharan
    @hbhasker
    There are two takeaways:
    • if you have a server on host and a guest in gvisor, then passing --network=host to runsc gives you 2x perf boost
      • if you have unix domain sockets inside runsc then the performance is tragic. 10x worse than anything else
    dammit wanted to reply inline
    FWIW --network=host also loosens your security of the sandbox because now you are using the host's networking stack
    so its a trade off
    Marek
    @majek
    I'm doing unix domain sockets on /tmp/pipe (within runsc container), am I expected to use another path for better performance?
    Bhasker Hariharan
    @hbhasker
    Uds is inside runsc is probably poor because it defaults to a 32kb buffer. Could you increase the buffer sizes and try
    Bhasker Hariharan
    @hbhasker
    @majek Could you share more data on how you ran these tests. I would like to run them locally to see if I can understand why UDS would perform badly
    Marek
    @majek
    @hbhasker at your service google/gvisor#5132
    Bhasker Hariharan
    @hbhasker
    Thanks! I will take a look and see if i can figure out why uds is performing much worse than tcp.
    stormgbs
    @stormgbs
    Hi, everybody, what kinds of workloads you are running with gVisor ? :)
    Robin Luk
    @lubinsz

    Hi, everybody, what kinds of workloads you are running with gVisor ? :)

    I think java applications are often used in gVisor.

    Pi dy/dx
    @pidydx
    Hello! I'm having sudden issues with gvisor using a configuration that I have had success with so I'm not sure what changed that is causing the problems. When I use crictl runp --runtime runsc sandbox.json as in the quickstart example I get "FATA[0000] run pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: dial unix /run/containerd/s/2935bdefdde6ca60c8f96d66dbc70beb464c3ca85ac649fcb98193a3b1e8d189: connect: connection refused: unknown"
    Travis DePrato
    @travigd
    Does gvisor disallow setuid? I have a legacy application that uses sudo that I'd like to run under gvisor, but I'm running into issues...
    Adin Scannell
    @amscanne
    @pidydx containerd 1.3.9 included a fix for a security issue that changed the paths for unix sockets: https://github.com/containerd/containerd/security/advisories/GHSA-36xw-fx78-c5r4 .. A compatible fix was included with 9eb77281c4fe1c1f252a0df67ca2c1fee8867b80
    We should send an announcement and update our containerd docs with this error (likely next week, or PRs accepted :)
    Adin Scannell
    @amscanne
    @travigd I believe that setuid is still on the backlog (bit ignored), if you file a bug it would be good to understand what requires sudo (and why) to inform the priority
    Pi dy/dx
    @pidydx
    @amscanne I am doing my testing with test-kitchen using ubuntu20.04 (which seems to have containerd 1.3.3) and I was having the problem with both 20201208.0 and 20201012.0 versions. Is that still likely related?
    Pi dy/dx
    @pidydx
    Huh, welp good news I guess. I swapped out to using containerd.io 1.4.* from docker repo and it all magically works as expected again
    Pi dy/dx
    @pidydx
    It is definitely a bit concerning that two different 1.3.3 packages in ubuntu20.04 appear to behave differently
    Hasan Turken
    @turkenh

    Hi everyone,
    Having an issue with gvisor on GKE.
    Running a pod with two containers and need them to communicate over Unix Domain Socket (a socket file on a shared emptydir volume). This works on non gvisor nodes but getting following on gvisor:

    "failed to create connection to unix socket: /tmp/socketfile.sock, error: dial unix /tmp/socketfile.sock: connect: connection refused"

    Any idea on how to make this work?
    Fabricio Voznika
    @fvoznika
    Shared volumes are mounted independently inside containers and Unix Domain Socket from one container will not be visible in the other container. Are you able to use abstract UDS?
    If not, it might work if the shared volume is tmpfs (medium: Memory). In this case, there will be a single volume mounted in the pod that is shared between containers.
    Hasan Turken
    @turkenh
    I did try medium: Memory but no luck.
    I don't know what abstract UDS is (and how to use it in a pod between two containers), but will check and try.
    Fabricio Voznika
    @fvoznika
    Abstract sockets uses a name that starts with "\0" followed by the socket name. The socket can be addressed by other processes using the same name, but it's not stored in the file system, thus it can be shared between containers of the same pod.
    Also, I'm almost sure that "medium: Memory" with VFS2. You can find instructions to enable VFS2 in GKE here: https://docs.google.com/document/d/1_qMNPzXyDwtGnZaTeQLMmZ2JhKwiTfjwtGG2Z5zA030/edit
    I can give it a try later today to confirm...
    Hasan Turken
    @turkenh
    Thanks @fvoznika
    I see VFS2 requires 1.18.9-gke.1501 or greater which is not possible for us ATM. But will keep this in mind for the future once we are able to move to that version. Is it planned to have it enabled by default on GKE at some point?
    Hasan Turken
    @turkenh
    wow, abstract sockets just worked!
    @fvoznika thanks a million!!
    Fabricio Voznika
    @fvoznika
    I'm glad it worked. As for VFS2, yes it will become default as soon as we finish validating it.