Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Benjamin Räthlein
    @raethlein
    interesting. we just tried to reproduce the error, basically performing the same steps you do, but we are able to login. Maybe it is a browser issue, Chrome and Safari are tested, or do you block cookies?
    Connor Goddard
    @cgddrd
    Hi @raethlein, apologies for the radio silence recently, we're plowing ahead with our trial of ML Hub - really good work! One thing I've noticed, the last few releases haven't had chart files included with the them? Is there any particular reason for this?
    Benjamin Räthlein
    @raethlein
    Hey @cgddrd, thanks a lot :)
    There were no breaking changes which would require a fundamentally new helm chart except for updating the default value of the hub image and to be honest, I was just too lazy to do it for the releases 😅(the default values can be overriden by passing a value.yaml file to the helm command). Though, I have added a chart to the latest release 0.1.10 :)
    Connor Goddard
    @cgddrd
    image.png
    Hey @raethlein - no worries, my bad, I should have realized! I'm now running 0.1.10, but have a couple of observations/issues?
    1. For some reason (compared to 0.1.7) I now have an '0' appear in between my first and surname from my GSuite username:
    (see above)
    In 0.1.7, this was instead just 'connorgoddard'
    I think 0.1.7 had the right behaviour.
    Secondly, I'm experiencing reliablilty issues with the Hub container in 0.1.10 - it seems to spontaneously die , and gives the error message "[W 2019-12-06 10:58:43.695 JupyterHub app:1805] Cannot connect to managed service cleanup-service at http://hub:9000"
    Connor Goddard
    @cgddrd
    I'm, not sure if I'm just missing something here (maybe config)? I can confirm this doesn't happen in 0.1.7 (so far as I can tell). I'm planning on testing 0.1.9 (I skipped straight from 0.1.7 to 0.1.10.
    Connor Goddard
    @cgddrd
    Further investigation here suggests this issue described above has been around since 0.1.8 where the cleanup service was introduced.
    Annotation 2019-12-11 084349.png
    When I list the available services for 0.1.8-0.1.10, I can't find anything related to a cleanup service?
    Not sure if there's some kind of configuration missing from the chart that is preventing the cleanup service from starting up?
    Connor Goddard
    @cgddrd
    OK, looking at your commits (which I should have done earlier) has shed some light on things...
    Username normalisation not an issue, I understand what's happening now
    I now also see the cleanup service is a Jupyterhub service, I've still got issues with this, but I can do a little more digging unless you guys have a quick solution?
    Shouldn't jumped straight to the commits in the first instance, sorry about that.
    Benjamin Räthlein
    @raethlein
    Hey @cgddrd , no worries, thanks a lot for investing your time! I will try to check whats going on with the cleanup service in kubernetes mode. Unfortunately, the last days of this year are very busy, so i am not sure when we can fix it.
    I dont like the new username normalization scheme either and its probably a quite unlike scenario, so we might change it back again with the possibility to configure it somehow.
    Simon Sander
    @sim-san
    I want to add an enviroment variable to the each juypter notebook. But I got the following error:
            c.Spawner.environment["TF_FORCE_GPU_ALLOW_GROWTH"] = "true"
        TypeError: 'LazyConfigValue' object does not support item assignment
    Benjamin Räthlein
    @raethlein
    Hey @sim-san , does c.Spawner.environment.update({"TF_FORCE_GPU_ALLOW_GROWTH": "true"}) work?
    Connor Goddard
    @cgddrd
    @raethlein, Merry Christmas and Happy New Year! Can I confirm, is the cleanup service meant to run as a seperate Kubernetes service, or is it meant to run as an internal service within the main Hub pod?
    Connor Goddard
    @cgddrd
    I'm struggling to get the SSH connectivity running in MLHub via Kubernetes. It looks like the script you download using 'Tools -> 'SSH' proxies the connection through the Proxy Pod via Port 80 (rather than Port 22), but no matter what I do, I can't get this to connect. Any ideas @raethlein ?
    Benjamin Räthlein
    @raethlein
    @cgddrd Thank you very much! Also Merry Christmas and Happy New Year to you :)
    The cleanup service should run inside the hub pod, but there seems to be an issue with it in Kubernetes mode, so I added the possibility to disable it via an environment variable ml-tooling/ml-hub@c80514e.
    With regards to ssh, do you run hub version 0.1.10? It contains a fix concerning ssh in the Kubernetes deployment.
    And do you have SSL enabled (hub is accessible via HTTPS)? Right now, ssh only works when HTTPS is enabled due to some proxy stuff in the hub
    Connor Goddard
    @cgddrd
    Hi @raethlein , many thanks for your reply. I think I've got a fix for the cleanup service in K8S - I'm putting together a PR as I write and I'll share shortly.
    Re
    Connor Goddard
    @cgddrd
    Thanks for your help regarding SSH. I realised I'd missed those instructions in the repo README (sorry about that!) I've confirmed enabling SSL does allow SSH to work in a LOCAL DOCKER ENVIRONMENT, however, I'm having issues getting SSL working under K8S. I've tried setting the 'START_SSL' environment variable to "True" within the config.yaml file (and have confirmed this has been enabled in the pod), but everytime I try to access MLHub over HTTPS, I get errors saying the SSL connection is invalid. I'm not sure if this is happening because I'm trying to test using the self-signed auto generated certs MLHub creates by default, or whether im missing something else (possibly I need to configure something for the proxy hub as well)?
    Any example configuration setups for using MLHub with HTTPS and Kubernetes would be very much appreciated.
    Connor Goddard
    @cgddrd
    Proxy pod not proxy hub
    Connor Goddard
    @cgddrd
    Hi @raethlein any thoughts on this?
    Benjamin Räthlein
    @raethlein

    Hey @cgddrd , adding this to the config.yaml should work

    proxy:  
      extraEnv:
        SSL_ENABLED: true
      https:
        hosts:
          - my.domain.com
        type: manual
        manual:
          key: |
            -----BEGIN RSA PRIVATE KEY-----
            ...
            -----END RSA PRIVATE KEY-----
          cert: |
            -----BEGIN CERTIFICATE-----
            ...
            -----END CERTIFICATE-----

    I also work on making the setup with the chart easier by separating the deployment configuration from the JupyterHub & Spawner

    Connor Goddard
    @cgddrd
    Many thanks @raethlein , so I'm guessing from your last post that it's not possible to use the auto-generated certs for MlHub with Kubernetes?
    Benjamin Räthlein
    @raethlein
    @cgddrd Actually I don't know whether or not it works as I have only tested the manual approach so far. I hope I will find some time soon to have a look at the autogenerated scenario
    Anton
    @Spar9a
    Hi, guys. How i can open admin panel ml-hub after installation?
    Using Windows 10 Pro, Docker installed, Kybernetes enabled.
    Command that using for install hub: docker run -p 8080 -v /var/run/docker.sock:/var/run/docker.sock -v jupyterhub_data:/data mltooling/ml-hub:latest
    Mike Nguyen
    @Darwiner

    Hi there. I'm trying to deploy ml-hub on k8s using https://github.com/ml-tooling/zero-to-mlhub-k8s. Things seem fine, up until the point where I actually try to start a server up, and, for some reason, I'm having issues with the creation of a pod? Anyone else run into any issues?

    Of note... I've done a deployment of https://github.com/jupyterhub/zero-to-jupyterhub-k8s and that works fine, not getting any issues at all on that front.

    19m Warning FailedScheduling pod/ws-admin-mlhub 0/7 nodes are available: 3 node(s) had taints that the pod didn't tolerate, 4 node(s) didn't find available persistent volumes to bind.

    Not sure what taint this is about, as my k8s nodes don't even have any taints (except for the masters).

    Benjamin Räthlein
    @raethlein
    Hey @Spar9a , if you go to localhost:8080 (or whatever port from the host was mapped; you can check it via docker ps -a or you delete the container again and start it via -p 8080:8080). There, you have to click on register, enter admin as the username and provide any password. Afterwards, you can log in. Does this work for you?

    Hey @Darwiner , I am sorry to hear about the inconvenience. Just as an update upfront: we pushed version 1.0.0 and updated the documentation where we simplified - or at least hope so ;) - the Kubernetes deployment by relying more on the JupyterHub-way of configuring things. In other words, we separated the JupyterHub configuration and the Kubernetes deployment configuration from each other. All feedback is very much appreciated :)

    With the error you reported, would you mind to share the configuration yaml you provide during the helm chart update?
    One issue could be that the mlhub-k8s project is based on an older version of jupyterhub-k8s, so there might be changes. I can try to have a look and figure out what the issue is.

    Guillaume G.
    @guillaumegauthier
    Hi there, while deploying mlhub on k8s, is there a way to set cpu and memory requests? Reading mlhubkubernetesspawner.py, it seems that these values are hard-coded to 0 (L84). If so, which configuration do you use to make sure the workspaces are properly distributed across the k8s nodes?
    Mateo Periago Serrano
    @MateoPeri
    Hi, just deployed on Ubuntu 20.04 with the most basic command. If I go to <ip>:8080, I get a 404. The only other text on screen is "openresty/1.15.8.1". There is clearly something listening as I got served the 404, but I see no login/registration page. ML Workspace works fine (on a different port).
    Benjamin Räthlein
    @raethlein

    Hey @MateoPeri , what is the command you used to deploy it? When I run

    docker run \
        -p 8080:8080 \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -v jupyterhub_data:/data \

    I can access it under 8080 (it redirects to localhost:8080/hub/login, so maybe you can try to append that path directly).
    Another possibility could be that the mlhub container did not start properly, so you could try to remove and re-create it.

    Simon Sander
    @sim-san

    I installed the ml-hub on a new server. Everything works fine. But when I create a new workspace, i got the following error in the ml-hub logs.

    traitlets.traitlets.TraitError: The 'ip' trait of a Server instance must be a unicode string, but a value of None <class 'NoneType'> was specified.

    Furthermore, the complete workspace info is crashed:

    grafik.png
    Benjamin Räthlein
    @raethlein
    @sim-san could you provide some more information, please? Which hub image do you use and which workspace?
    Simon Sander
    @sim-san
    I using the hub image 0.1.10 and the workspace image 0.9.1. But I forked the hub image to add further features and I add also a jupyterhub_user_config.py, further I update some libaries in the Dockerfile of the workspace. I know that my changes make it very difficult for you to reproduce the error. Could you give me a hint, how this error can occur.
    Simon Sander
    @sim-san
    @raethlein I have found my mistake. I have selected a GPU with an ID that is not present on the system. As a result the GPU could not be found and an error occurred which is expressed in the described behavior.
    @raethlein I have a further question. We are hosting our ml-workspace images on a private docker registry with authentication. If the images are not already pulled on the host, the following error message will appear:
    MicrosoftTeams-image (1).png
    Simon Sander
    @sim-san
    Do you know if it is possible to add authentication to the ml-hub? I couldn't immediately find a place in the code where this would be easily possible.
    Benjamin Räthlein
    @raethlein
    @sim-san great that you found the issue! Do you use ml-hub in local mode (single server)? Then I think the way to go would be to manually perform docker login ... on the host. That said, I have not spent too much time on this yet. With regards to the other features you have added to Hub, if they are generally relevant feel free to create a Pull Request :)
    fiveight58
    @fiveight58
    ml-hub is a greate project!Thanks. I will deploy this in my k8s .