Translate and then memmap.File.MapInternal and read from the returned BlockSeq
This is what I'm trying to do here: google/gvisor#6397
Hello! I’m struggling to get gVisor to work in Kubernetes v1.21 after I upgraded containerd to v1.5.5 and gVisor to 20210720 on my worker nodes. Looking at the containerd logs, I see the following message:
level=error msg="RunPodSandbox for &PodSandboxMetadata{...} failed, error" error="failed to create containerd task: failed to create shim: type with url runtimeoptions.v1.Options: not found: unknown"
level=error msg="failed to delete" cmd="/usr/bin/containerd-shim-runsc-v1 ..."I can use ctr to manually run a container with runsc, so I think gVisor is installed correctly. Any idea what the issue could be? Thanks!
7 replies
Hello there. I am trying to use cyclictest to benchmark the CPU latency in gVisor containers:docker run --privileged -it --runtime=runsc --cpus=4 --memory="8192m" leap-runsc /bin/bash
I am getting:
Unable to change scheduling policy!
either run as root or join realtime group
Apparently I am running a privileged container, logged into the shell with root user. There are no issues with runc, kata-runtime etc. Is there a way to run cyclictest in gVisor?
In gVisor, sentry intercepts all app syscalls, including all sched_* syscalls., see https://github.com/google/gvisor/blob/master/pkg/sentry/syscalls/linux/sys_sched.go. And sentry reuses the scheduler in go runtime (which is a work-steal, FCFS scheduler). So basically, there's no special sched features, like policy, priority.Unable to change scheduling policy! either run as root or join realtime group
Hi guys, I don't know much about Linux syscall. I am trying to intercept any disk io syscall from a container to OS kernel, and I found out about this awesome gVisor project. After reading its architecture for a while, I am wondering why we need Gofer for handling disk i/o, why Sentry can not handle it?
Any explanation would be appreciated!
Hi guys, I don't know much about Linux syscall. I am trying to intercept any disk io syscall from a container to OS kernel, and I found out about this awesome gVisor project. After reading its architecture for a while, I am wondering why we need Gofer for handling disk i/o, why Sentry can not handle it?
Any explanation would be appreciated!
The reason lies in the design of gVisor. gVisor sandbox process shall be reduced in small amount of syscalls (into host kernel); and some file-related syscalls are dangerous but indispensable. So gVisor uses the way of defense-in-depth. Put those syscalls into gofer process, and let sandbox process to talk with gofer processes in a protocol (p9 in current implementation). Actually, not all file-related syscalls are put into gofer, as this path is too slow. So gofer process may donate FDs into sandbox process, and it calls pread/pwrite to make sure some data paths are faster.
Thanks for the reply @tanjianfeng !
I have more questions. I tried to intercept some write(2) system calls issued by a running container. For a starter, I added fmt.Println("log") in https://github.com/google/gvisor/blob/master/pkg/sentry/syscalls/linux/sys_write.go#L46 to make sure I intercept at the right location, as what Ian Lewis said in https://groups.google.com/g/gvisor-users/c/r4xrbpP9-pQ. It works well, whenever there is any write system call, my write log is triggered.
However, when I tried to do more complex logic (I tried to make an HTTP request when Sentry detects writes syscall to a specific file, I tried to log the write requests in a file using os.WriteFile), the running container stop immediately.
Is this because of the limited syscall that Sentry can use? so I can't create an HTTP request or write a file from Sentry? Is there any better way to intercept file IO syscall in Sentry?
Thank you for the guidance!
You already can intercept the write syscall, and the problem you face now is how to redirect the write content to "somewhere".
When you use fmt.Println("log"), the content is rewritten into stdout or stderr of sandbox process (when you enable debug option, it's in runsc log file like *.boot or a coalescent log file).
If you want to rewrite the content to some specified file, you need to borrow a "fd" from gofer. The background behind this, sentry does not open file, it lets gofer to open file, and donate fd back to gofer. To rewrite to the given fd, you can refer this function: https://github.com/google/gvisor/blob/master/pkg/sentry/fsimpl/gofer/handle.go#L107
make runsc
bazel-bin.
$ stat bazel-bin
File: bazel-bin -> /root/.cache/bazel/_bazel_root/06dac7d55a426239d70c12d704d4fc29/execroot/__main__/bazel-out/k8-opt/bin
Size: 102 Blocks: 8 IO Block: 4096 symbolic link
Device: 807h/2055d Inode: 562602 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-09-09 22:17:37.958164085 +0200
Modify: 2021-09-09 22:16:33.471035217 +0200
Change: 2021-09-09 22:16:33.471035217 +0200
Birth: -sudo cp ./bazel-bin/runsc/linux_amd64_pure_stripped/runsc /usr/local/bin command because it cannot access the path of bazel-bin/...
You already can intercept the write syscall, and the problem you face now is how to redirect the write content to "somewhere".
When you use fmt.Println("log"), the content is rewritten into stdout or stderr of sandbox process (when you enable debug option, it's in runsc log file like *.boot or a coalescent log file).
If you want to rewrite the content to some specified file, you need to borrow a "fd" from gofer. The background behind this, sentry does not open file, it lets gofer to open file, and donate fd back to gofer. To rewrite to the given fd, you can refer this function: https://github.com/google/gvisor/blob/master/pkg/sentry/fsimpl/gofer/handle.go#L107
Thanks for the recommendations!
I am stumbling upon the performance overhead of gVisor, if we use ptrace to intercept any syscalls, the syscall becomes ~20x slower (src: https://gvisor.dev/performance/syscall.csv), however why gVisor can still achieve low bandwidth overhead for file I/O as presented in https://gvisor.dev/performance/fio.csv?
Could somebody explain the reasoning behind this?
Thanks!