minrk on 1.4.1
minrk on 1.4.x
Backport PR #3436: ci: github w… Backport PR #3452: Fix document… Backport PR #3437: patch base h… and 7 more (compare)
cull-idleto timeout on the
GET /usersrequest but jupyterhub-idle-culler uses tornado which defaults
request_timeoutto 20 seconds and the API is taking about 50 seconds in some cases. Is there an easier way to set the request timeout via environment variable or are we stuck with setting
max-time = 60in $HOME/.curlrc?
My z2jh GKE cluster stalled badly under fairly mild load today, giving "serviice refused" errors. The autohttps pod restarted many times. Last -p log on the
secret-sync container has:
2021-05-10 09:29:19,247 INFO /usr/local/bin/acme-secret-sync.py watch-save --label=app=jupyterhub --label=release=jhub --label=chart=jupyterhub-0.11.1 --label=heritage=secret-sync proxy-public-tls-acme acme.json /etc/acme/acme.json 2021-05-10 09:30:24,876 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff5883de2b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/jhub/secrets/proxy-public-tls-acme
Any ideas where I could look for the problem?
kubectl logs --previous autohttps-9fdcfc86c-9jdwx secret-sync- the other containers in that pod give e.g.
previous terminated container "traefik" not found. Traefik image reported as
traefik:v2.3.7. Is there any other way I can diagnose the pod restart?
kubectl describe pods autohttps-9fdcfc86c-9jdwxshows that, of the two containers:
secret-synchas restarted 42 times,
traefik0 times. I didn't look specifically, but when the service went down, it seemed to me that there were many new restarts in the
autohttpspod - and therefore, I now see, the
secret-synccontainer. Is there anything I can or should do to mitigate?
@metric-chicken it is a bit tiresome, but you probably need to use a initContainer that
chown the folder of relevance before the normal user-server container mounts it.
singleuser.uid. The base user that the filesystem mounts as, when using NFS, currently: no. The base user that ends up running the container, yes - by starting as root, using a docker-stacks based image, and setting
NB_USERIDenvironment variable or something like that and it will during startup scripts, (which you may need to manually specify), could get it done.