Hi, I've been playing around with gvisor on an EKS cluster and have the following setup:
1) An EKS cluster with gvisor installed on every node.
2) A runtime class with the following definition:
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runscUsing that setup I can create deployments running with gvisor by specifying the runtime class in the spec like: runtimeClassName: gvisor.
This all works fine and I am able to get inside the pods and confirm that they are running with gvisor correctly.
The problem, however, is that if I specify resource limits for the containers, they are never enforced.
They work fine with the default runtime class and when a pod exceeds them it gets terminated, but with gvisor the pods exceed them and can even crash the whole node.
I've read that all containers should have limits specified and I am running a single container, but the limits still don't get enforced.
Any suggestions?
kubectl top pod and kubectl describe node/{NODE_WHERE_POD _IS_RUNNING} say?
@zkoopmanskubectl top pod:
NAME CPU(cores) MEMORY(bytes)
traffic-5879846794-kgb2m 382m 353Mikubectl describe node/...:
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
dmarinov traffic-5879846794-kgb2m 10m (0%) 10m (0%) 10Mi (0%) 10Mi (0%) 2m6s
kube-system aws-node-btl8z 25m (1%) 0 (0%) 0 (0%) 0 (0%) 2d19h
kube-system kube-proxy-c9nl2 100m (6%) 0 (0%) 0 (0%) 0 (0%) 2d19h
kube-system metrics-server-847dcc659d-mb59d 100m (6%) 0 (0%) 200Mi (17%) 0 (0%) 5m21s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 235m (14%) 10m (0%)
memory 210Mi (18%) 10Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0Let me know if you need any additional information.
the changing platform is for docker. How about containerd ?
Heya! A bit of a newbie question: In "The true cost of containing" (https://www.usenix.org/system/files/hotcloud19-paper-young.pdf) and the Performance guide (https://gvisor.dev/docs/architecture_guide/performance/) there are references to "internal" or "sandbox-internal" tmpfs filesystems. If I mount a tmpfs through docker's tmpfs option, would that be internal or external?
In that case, it'll be handled like bind mount, will go into the heavy path (sync with kernel through gofer).
2 replies
Runtimes: runc runsc io.containerd.runc.v2 io.containerd.runtime.v1.linuxThe path in
daemon.json is correct. But when I start a container with runtime: runsc it seems to loop with no log. Any clue ?
runsc do echo "hello world" show anything at all?
1 reply
creating container: cannot create sandbox: cannot read client sync file: waiting for sandbox to start: EOF
You're positive
runsc --debug --debug-log=/tmp/runsc/ do echo "hello world" doesn't create logs in /tmp/runsc/?If that is still the case you should do a stack trace as instructed in the debugging guide
1 reply
eric@vault:~/hauk $ uname -a
Linux vault 5.15.61-v8+ #1579 SMP PREEMPT Fri Aug 26 11:16:44 BST 2022 aarch64 GNU/Linux
Heya! A bit of a newbie question: In "The true cost of containing" (https://www.usenix.org/system/files/hotcloud19-paper-young.pdf) and the Performance guide (https://gvisor.dev/docs/architecture_guide/performance/) there are references to "internal" or "sandbox-internal" tmpfs filesystems. If I mount a tmpfs through docker's tmpfs option, would that be internal or external?
Good question - I'd like to know this too. Seeing less than ideal tmpfs performance when using /tmp to build Go binaries on an otherwise read-only filesystem
