Ask quick dev questions about JupyterHub, the multi-user server for Jupyter notebooks. Use discourse.jupyter.org for user questions, support, and discussion.
Then you probably want to subclass and define refresh_user as appropriate.
I don’t believe any Oauthenticator class defines refresh_user yet, though most probably should.
(Issue for oauthenticators implementing refresh_user: jupyterhub/oauthenticator#398 )
oauth_token_expires_in governs how often the single-user server must re-auth with the Hub, so you could set that that could be your access token expiry
Expiry, etc. are configured separately
I started working on a doc covering the various steps in jupyterhub/jupyterhub#3444
any idea why ?
We have over 5300 users in a jupyterhub DB right now which is causing
cull-idle to timeout on the GET /users request but jupyterhub-idle-culler uses tornado which defaults request_timeout to 20 seconds and the API is taking about 50 seconds in some cases. Is there an easier way to set the request timeout via environment variable or are we stuck with setting max-time = 60 in $HOME/.curlrc?
7 replies
My z2jh GKE cluster stalled badly under fairly mild load today, giving "serviice refused" errors. The autohttps pod restarted many times. Last -p log on the secret-sync container has:
2021-05-10 09:29:19,247 INFO /usr/local/bin/acme-secret-sync.py watch-save --label=app=jupyterhub --label=release=jhub --label=chart=jupyterhub-0.11.1 --label=heritage=secret-sync proxy-public-tls-acme acme.json /etc/acme/acme.json
2021-05-10 09:30:24,876 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff5883de2b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/jhub/secrets/proxy-public-tls-acmeAny ideas where I could look for the problem?
What version of traefik is used?
Why did the pod restart?
(kubectl logs --previous)
Sorry - that output was from a call to
kubectl logs --previous autohttps-9fdcfc86c-9jdwx secret-sync - the other containers in that pod give e.g. previous terminated container "traefik" not found. Traefik image reported as traefik:v2.3.7. Is there any other way I can diagnose the pod restart?
Does pods restart, or containers? Check the containerststuses - perhaps only secretsync container restarted, and perhaps that made the pod not be considered ready for a while perhaps (if it would be totally fine if it was unavailable for a long time btw - it shouldnt habe a readinessprobe)
Aha - yes - how helpful:
kubectl describe pods autohttps-9fdcfc86c-9jdwx shows that, of the two containers: traefik and secret-sync, secret-sync has restarted 42 times, traefik 0 times. I didn't look specifically, but when the service went down, it seemed to me that there were many new restarts in the autohttps pod - and therefore, I now see, the secret-sync container. Is there anything I can or should do to mitigate?
It is a problem if the pod isnt ready during that process though
Then no network traffic is accepted
But the container is only relevant on startup
Thanks @consideRatio for the suggestions.
Question about Extension Manager - I'm gathering that since JupyterLab runs as a non-privileged user in a JupyterHub environment, it should be expected that Extension Manager doesn't work? To the user, it says "Error communicating with server extension" with "TypeError: Failed to fetch". And even if it did work, extensions installed this way wouldn't persist between notebook pod starts?
4 replies
i want to have a baseline pandas and stuff like that for everyone - my google fu is weak this morning ... i have tested installing via conda in the base profile, as well as in the python profile... i thought and restarted jupyterhub services... i can install it per user, just trying to onboard folks with a baseline
@alibama: you are using the UNIX users to authenticate right?
2 replies
I've run into a potential problem and maybe someone can help. How do I handle user permissions with shared storage resources (i.e. shared drives). I have ztjh up and running. I'm mounting individual directories using NFS. I'm also mounting a shared volume using NFS. How would/could I set the owner of files to be $JUPYTERHUB_USER?
@metric-chicken it is a bit tiresome, but you probably need to use a initContainer that chown the folder of relevance before the normal user-server container mounts it.
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/421#issuecomment-725429444
@consideRatio I can probably make that work. Is there away to change the base user to something else? For instance, to something like user 'foo'? I'm migrating from a docker installation and I'm trying to keep the current user permissions working in the shared directory.
The base user, as is the user that the container starts as? Yes
singleuser.uid. The base user that the filesystem mounts as, when using NFS, currently: no. The base user that ends up running the container, yes - by starting as root, using a docker-stacks based image, and setting NB_USERID environment variable or something like that and it will during startup scripts, (which you may need to manually specify), could get it done.
Two things, first, it looks like this page moved https://github.com/berkeley-dsep-infra/datahub/blob/21e4a45c9f694578ec297c2947a0537f3bdcaa5b/deployments/data100/config/common.yaml#L65 does anyone have the new address? @yuvipanda I think you sent that to me a while back. Second, all the packages the users install are wiped out when the user's pod is restarted. How do I make them persist?
If I create a user env from a notebook it shows up in the list of kernels. However, if I stop and then start the server, the kernel is not there. I am using 'python -m ipykernel install --user --name myenv --display-name "Python (myenv)"' to create the env. If I run the same command from a terminal, the kernel persists after I restart the server but all of the packages I installed are missing. I have added the "envs_dirs:" to the .condarc and point it to a directory in the users dir on an nfs share.
Any ideas?