Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 06:59
    yuvipanda review_requested #3413
  • 01:52
    kafonek commented #3403
  • 01:50
    kafonek edited #3403
  • 01:41
    kafonek synchronize #3403
  • 01:40
    kafonek synchronize #3403
  • Apr 11 21:38
    Hoeze edited #2986
  • Apr 11 21:37
    Hoeze edited #2986
  • Apr 11 21:35
    Hoeze commented #2986
  • Apr 10 18:28
    welcome[bot] commented #3414
  • Apr 10 18:28
    olifre opened #3414
  • Apr 10 00:52
    willingc commented #2726
  • Apr 10 00:44
    willingc review_requested #2726
  • Apr 10 00:44
    willingc review_requested #2726
  • Apr 09 21:14
    naatebarber synchronize #3413
  • Apr 09 21:07
    naatebarber opened #3413
  • Apr 09 15:59
    kafonek edited #3403
  • Apr 09 15:02
    kafonek synchronize #3403
  • Apr 09 14:24
    minrk commented #3412
  • Apr 09 13:52
    minrk commented #3397
  • Apr 09 13:51
    minrk commented #3393
yuvipanda
@Yuvi:matrix.org
[m]
what happens if the oauth token never expires?
Min RK
@minrk
It stays in the db forever
It doesn't make sense for oauth token max age to exceed cookie max age
since that's where it's stored
which is 2 weeks by default
yuvipanda
@Yuvi:matrix.org
[m]
right, that makes sense. that is what i was looking for too - confused since the cookie max age hasn't been reached
Min RK
@minrk
Yeah, this all boils down to: I never set or even really thought about oauth token max age, so we've been using oauthlib's defaults, mostly by accident.
yuvipanda
@Yuvi:matrix.org
[m]
:D
what would be the advantage of having oauth token max age be shorter than cookie age?
Min RK
@minrk
Probably isn't really one, other than cookie max age (which governs hub session duration) is usually longer than server max age, which is the upper limit for a reasonable oauth token expiry
yuvipanda
@Yuvi:matrix.org
[m]
server max age is how long a server can run?
Min RK
@minrk
yeah, if set
Maybe we should even make the default use cookie_max_age_days
yuvipanda
@Yuvi:matrix.org
[m]
yeah, what would be the disadvantage of that?
Min RK
@minrk
I don't think there is one
yuvipanda
@Yuvi:matrix.org
[m]
yay! What do you think of doing that in your PR?
Min RK
@minrk
Just pushed it
yuvipanda
@Yuvi:matrix.org
[m]
w00t!
@minrk: I'll merge it once tests pass
to recap: API tokens are sent to the python server process. OAuth tokens are stored in the client as cookies. The cookie expiry time didn't match oauth token expiry time, and now it does.
Min RK
@minrk
Yes
Let's maybe try to push out hub 1.4 next week
yuvipanda
@Yuvi:matrix.org
[m]
yeah
@minrk: merged! \o/
Matt Kafonek
@kafonek
@minrk @Yuvi:matrix.org is it accurate to say the tokens issued to services could also be called "client_secret" instead? For anyone coming from a Keycloak background, that might help
yuvipanda
@Yuvi:matrix.org
[m]
yeah that's more standard oauth terminology. Thoughts, minrk (Min RK)?
@kafonek: can you open an issue as well?
Matt Kafonek
@kafonek
yep one moment
Dan Parsons
@danparsons_twitter
I am trying desperately to get my jupyterhub on EKS installation to stop creating an ELB that has port 80 open. I've tried every helm value I can find. Nothing works. Anyone been through this before?
Dan Parsons
@danparsons_twitter
So, I solved my above problem by putting jupyterhub behind ingress-nginx. Now http->https and https work great. However, now websockets fail. It can't connect to the kernel now. "A connection to the notebook server could not be established.". I've dug through logs, googled for hours, tried a bunch of different nginx annotations, nothing is fixing it.
This guy seems like he's figured it out, but the nginx config he links to isn't helpful to me, mapping it into kubernetes (eks): jupyterhub/jupyterhub#3017
Any help anyone can offer on troubleshooting or how to fix this is greatly appreciated. All I see in the hub pod log is 'Replacing stale connection' over and over again.
Angus Hollands
@agoose77:matrix.org
[m]

RE the nginx config, I'd assume that it's related to the ws forwarding settings. Not hugely familiar with the intricacies of it all though.

RE JupyterHub,perhaps you want https://zero-to-jupyterhub.readthedocs.io/en/stable/resources/reference.html#hub-service-type or proxy-service-type?

Dan Parsons
@danparsons_twitter
What I ended up having to do, to fix the websocket/kernel issue, was annotating my ingress controller's Service with "service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp" (instead of the default http). That made everything work. But it also prevents me from seeing real source IP addresses anymore. Which sucks because I need to IP restrict access to this resource. So I ended up having to make a dedicated ingress controller installation just for Jupyterhub, so I can IP restrict at the Ingress controller level (via spec:LoadBalancer:loadBalancerSourceRanges), and gave the ingress controller a different ingress class so I can still have a normal behaving ingress controller on this cluster. Could not use Jupyterhub's built in IP whitelisting feature because jupyterhub no longer sees real external IP addresses.
I could not use the zero-to-jupyterhub stock configuration because guess what, it doesn't support http to https redirection. Common knowledge but, you can't do that on Kube with a regular LoadBalancer. I'm surprised zero-to-jupyterhub doesn't cover that. It does cover using a custom ingress controller, briefly, but totally omits the part where you have to use TCP as the backend protocol (instead of HTTP) or, well, Jupyterhub doesn't work at all.
yuvipanda
@Yuvi:matrix.org
[m]
hi @danparsons_twitter. If you're using z2jh with the default autohttps setup, it does do http -> https redirect
curl -v http://datahub.berkeley.edu gives me a 301 to https
can you help me understand what is missing?
also I did not know JupyterHub had a way to allow specific IPs only to talk to it...
Dan Parsons
@danparsons_twitter
Yeah, I'm not using default autohttps. Why would I use letsencrypt when I'm on AWS EKS and can use ACM certs? I also wanted to terminate TLS at the load balancer. I don't know how datahub.berkeley.edu is doing http->https redirect, but I can tell you they're not doing it with a plain old kubernetes LoadBalancer - which is how z2jh tells you to set things up
You must use an ingress controller if you want http to https redirect. And if you use an ingress controller, you have to configure its fronting ELB to use tcp, not http, for the backend, in order for the websocket connections to work. (z2jh mentions using an ingress controller briefly but does not tell you this key part in making it work)
Berkeley could have a totally different non-kubernetes load balancer in front of their jupyterhub that handles http->https redirect for them, I don't know, I only know about my setup
yuvipanda
@Yuvi:matrix.org
[m]
@danparsons_twitter: aaah, I see. I think using autohttps is the most common way to deploy https with z2jh, and that's the recommended way as well. ACM is considered a somewhat advanced use case, precisely for the things you mention - ther's some documentation under https://zero-to-jupyterhub.readthedocs.io/en/stable/administrator/security.html#off-loading-ssl-to-a-load-balancer. jupyterhub/zero-to-jupyterhub-k8s#1811 is probably related to your issues, would be great if you can comment on that
anyway, tldr is that you should use autohttps for the smoothest possible experience :)
but we'd love contributions to make the offload situation easier!!! If you can even write down your experiences there, that would be very helpful, @danparsons_twitter
the recommended loadbalancer way is loadbalancer + autohttps, which is what berkeley's datahub uses.
@danparsons_twitter: plus, if you can open an issue about needing to add service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp, what would be helpful to debug too. I haven't needed to do that in the past with EKS.
yuvipanda
@Yuvi:matrix.org
[m]

I'm also very curious about what you mean by

Could not use Jupyterhub's built in IP whitelisting feature because jupyterhub no longer sees real external IP addresses.

Anyway, I'm sorry you didn't have the best experience possible, and hope that can be used to make experiences better in the future for others trying to do similar things as you
there's also a bunch more issues in https://github.com/jupyterhub/zero-to-jupyterhub-k8s/search?q=offload&type=issues that might be relevant to you