Ask quick dev questions about JupyterHub, the multi-user server for Jupyter notebooks. Use discourse.jupyter.org for user questions, support, and discussion.
It doesn't make sense for oauth token max age to exceed cookie max age
since that's where it's stored
which is 2 weeks by default
right, that makes sense. that is what i was looking for too - confused since the cookie max age hasn't been reached
what would be the advantage of having oauth token max age be shorter than cookie age?
Maybe we should even make the default use cookie_max_age_days
@minrk: I'll merge it once tests pass
to recap: API tokens are sent to the python server process. OAuth tokens are stored in the client as cookies. The cookie expiry time didn't match oauth token expiry time, and now it does.
Let's maybe try to push out hub 1.4 next week
@minrk: merged! \o/
yeah that's more standard oauth terminology. Thoughts, minrk (Min RK)?
@kafonek: can you open an issue as well?
So, I solved my above problem by putting jupyterhub behind ingress-nginx. Now http->https and https work great. However, now websockets fail. It can't connect to the kernel now. "A connection to the notebook server could not be established.". I've dug through logs, googled for hours, tried a bunch of different nginx annotations, nothing is fixing it.
This guy seems like he's figured it out, but the nginx config he links to isn't helpful to me, mapping it into kubernetes (eks): jupyterhub/jupyterhub#3017
Any help anyone can offer on troubleshooting or how to fix this is greatly appreciated. All I see in the hub pod log is 'Replacing stale connection' over and over again.
RE the nginx config, I'd assume that it's related to the ws forwarding settings. Not hugely familiar with the intricacies of it all though.
RE JupyterHub,perhaps you want https://zero-to-jupyterhub.readthedocs.io/en/stable/resources/reference.html#hub-service-type or proxy-service-type?
What I ended up having to do, to fix the websocket/kernel issue, was annotating my ingress controller's Service with "service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp" (instead of the default http). That made everything work. But it also prevents me from seeing real source IP addresses anymore. Which sucks because I need to IP restrict access to this resource. So I ended up having to make a dedicated ingress controller installation just for Jupyterhub, so I can IP restrict at the Ingress controller level (via spec:LoadBalancer:loadBalancerSourceRanges), and gave the ingress controller a different ingress class so I can still have a normal behaving ingress controller on this cluster. Could not use Jupyterhub's built in IP whitelisting feature because jupyterhub no longer sees real external IP addresses.
I could not use the zero-to-jupyterhub stock configuration because guess what, it doesn't support http to https redirection. Common knowledge but, you can't do that on Kube with a regular LoadBalancer. I'm surprised zero-to-jupyterhub doesn't cover that. It does cover using a custom ingress controller, briefly, but totally omits the part where you have to use TCP as the backend protocol (instead of HTTP) or, well, Jupyterhub doesn't work at all.
hi @danparsons_twitter. If you're using z2jh with the default autohttps setup, it does do http -> https redirect
curl -v http://datahub.berkeley.edu gives me a 301 to https
can you help me understand what is missing?
also I did not know JupyterHub had a way to allow specific IPs only to talk to it...
Yeah, I'm not using default autohttps. Why would I use letsencrypt when I'm on AWS EKS and can use ACM certs? I also wanted to terminate TLS at the load balancer. I don't know how datahub.berkeley.edu is doing http->https redirect, but I can tell you they're not doing it with a plain old kubernetes LoadBalancer - which is how z2jh tells you to set things up
You must use an ingress controller if you want http to https redirect. And if you use an ingress controller, you have to configure its fronting ELB to use tcp, not http, for the backend, in order for the websocket connections to work. (z2jh mentions using an ingress controller briefly but does not tell you this key part in making it work)
Berkeley could have a totally different non-kubernetes load balancer in front of their jupyterhub that handles http->https redirect for them, I don't know, I only know about my setup
@danparsons_twitter: aaah, I see. I think using autohttps is the most common way to deploy https with z2jh, and that's the recommended way as well. ACM is considered a somewhat advanced use case, precisely for the things you mention - ther's some documentation under https://zero-to-jupyterhub.readthedocs.io/en/stable/administrator/security.html#off-loading-ssl-to-a-load-balancer. jupyterhub/zero-to-jupyterhub-k8s#1811 is probably related to your issues, would be great if you can comment on that
anyway, tldr is that you should use autohttps for the smoothest possible experience :)
but we'd love contributions to make the offload situation easier!!! If you can even write down your experiences there, that would be very helpful, @danparsons_twitter
the recommended loadbalancer way is loadbalancer + autohttps, which is what berkeley's datahub uses.
@danparsons_twitter: plus, if you can open an issue about needing to add
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp, what would be helpful to debug too. I haven't needed to do that in the past with EKS.
I'm also very curious about what you mean by
Could not use Jupyterhub's built in IP whitelisting feature because jupyterhub no longer sees real external IP addresses.
Anyway, I'm sorry you didn't have the best experience possible, and hope that can be used to make experiences better in the future for others trying to do similar things as you
there's also a bunch more issues in https://github.com/jupyterhub/zero-to-jupyterhub-k8s/search?q=offload&type=issues that might be relevant to you