Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
Repo info
    Dan Lester
    jupyter labextension install @ideonate/jupyter-containds
    Or you can search within the extension manager in JupyterLab. I'll update the docs.
    Axel Larsson
    Hey! I am testing out containDS on our company hub, it's very promising! I've had some minor issues, but solved them all so far. One thing that I'd like to do is to restrict non-technical users so that they can only launch already created voila dashboards, or at the very least remove their ability to launch a "normal server". I was playing around with DockerSpawner.allowed_images/image_whitelist, however, since it's the same image for both the normal server and the dashboard servers, just different commands, it won't do. Do you have any suggestions for how to proceed?
    Dan Lester
    @AxelTLarsson thanks for your feedback! Yes, I think this is a good suggestion. The main question is how to convenient assign 'non-technical' versus 'technical' status.
    Easiest (to start at least) would be an allow/block list to name dashboard creators
    Better would be to add to JupyterHub Groups - but there is no UI for that at the moment so I would want to add one myself really
    And then how to reshape the JupyterHub UI for these non-technical users so that, for example, ideally they don't see a 'My Server' or 'New Dashboard' button at all, and only get to see the list of dashboards.
    Some of this functionality would sit in JupyterHub itself ideally, but doesn't really make sense to the core JupyterHub project...
    Anyway, this is something I want to work on. All ideas and feedback welcome!
    I have created an issue to track ideonate/cdsdashboards#37
    Axel Larsson

    I haven't found that much info on JupyterHub Groups, as far as I understand it though, there are two groups "admin", and non-admin. Or, rather, there is perhaps only one group "admin". Perhaps one could add a new group - "dashboard-users", whose users would only be allowed to list/launch already existing dashboards, and not a "normal" server. The UI would then reflect your group belonging, as it does with showing the "Admin" tab already.
    However, for starters, I would be perfectly fine with some slightly hacky way to throw an error at the non-techies, if they disregard my instructions and actually click these "My Server".
    What I'm thinking right now is to experiment with passing in the user name to the notebook server (Docker container), and then have a hook that checks a hard-coded allowed list of users, e.g. in /usr/local/bin/before-notebook.d, and if not allowed, refuses to start, somehow... It won't be pretty, but might do for now.

    I am following the issue with great interest, and I definitely think it could be a very useful feature for many.

    Dan Lester
    Actually 'admin' is just a flag that is assigned to each user (True/False). Groups are a fairly generic JupyterHub concept, but aren't really used anywhere... (other than in ContainDS to control access to Dashboards)
    Without changing JupyterHub core, the UI change might be easy enough, but then ideally would also be reinforced with something like you've suggested if they do attempt to use the API to start a server anyway.
    Sid Feygin
    Groups would be essential for our use case as well, which I think is somewhat different from the above. Basically, we will have internal data scientists who are dashboard creators and external users (clients) who will be consuming the dashboards and not permitted to run code. Ideally, authorization to view dashboards could be assigned on a per-group basis and be well-integrated with our JupyterHub-compatible OAuth provider of choice (AWS Cognito). I can definitely make more progress towards this end once ideonate/cdsdashboards#37 is resolved and jupyterhub/jupyterhub#3133) is merged. Any other significant challenges to implementing this solution that would require changes to either containds or JupyterHub? Happy to help out if so.
    Victor Tolpegin
    I'd love to see groups integrated with dashboards as well. I've loved using the dashboards feature thus far, and I know that integrating groups would provide us additional value (both in dashboards, and in jupyter overall)
    Dan Lester
    Thanks all. I think @sfwatergit and @AxelTLarsson would benefit from the same kind of approach, maybe needing some customization as well. @DarkmatterVale thank you for your encouragement too!
    I will work on something based on JupyterHub groups that at least gives a reasonable UI and passes something to spawners or servers so they know if that user should be allowed to spawn or not.
    Victor Tolpegin
    Thank you! And thanks for all of your work building out the dashboards feature. It's awesome!
    Dan Lester
    @DarkmatterVale Great to hear you like it!
    Axel Larsson

    Thank you so much for this project, it's going to improve my quality of life at work, immensely, I'm sure!
    I was able to implement a hack to disallow "non-technical" users from launching anything but dashboard servers, by adding a script to /usr/local/bin/before-notebook.d/disallow-non-techies.sh like this:

    function assert_allowed_user() {
        if [[ ! " ${ALLOWED_NB_USERS[@]} " =~ " ${GITHUB_USER} " ]]; then
            # whatever you want to do when array doesn't contain value
            echo "${GITHUB_USER} is not allowed to start a normal notebook server"
            exit 1
    if [ -z ${GITHUB_USER+x} ]; then
        # If $GITHUB_USER is not set, we are attempting to start a voila (dashboard) server
        # => go ahead
        echo "Allowing dashboard start"
        # If GITHUB_USER is not set, we are starting a normal notebook server, check if that is ok

    It's not very pretty, and relies on the presence of $GITHUB_USER to determine if it's launching a normal server or a dashboard. So far, it seems to work alright, but I'm eagerly awaiting any development that will obviate this hack.

    Dan Lester
    @AxelTLarsson thank you for your kind words, and for sharing this solution. I have made some interesting experimental commits to master if anyone wants to take a look.

    In your config add:
    c.CDSDashboardsConfig.spawn_allow_group = 'spawners-group'
    (or whatever named group you want, it will be created if it doesn't exist)
    this group will be the only people allowed to spawn servers or create dashboards.

    Alternatively use:
    c.CDSDashboardsConfig.spawn_block_group = 'viewers-group'
    to specify a list of users who should only be viewers - useful if most new users are developers.

    Dan Lester
    This roughly restricts the UI so they can't do any spawning or dashboard creation, except to do this there is now a custom /hub/home-cds page instead of /hub/home.
    If they go to /home directly (or /hub/spawn/dan for that matter) then they can attempt to start a server but it will throw an error straight away.
    I have also added a basic Group management UI to make it easier to add/remove users from the groups. In @sfwatergit 's case, we would also want to sync to the group inside the Authenticator (much is it gets admin status at the moment) - let's take a look.
    Axel Larsson
    Wow, very nice @danlester! I just tried it out and it works very well. It does seem that I have to explicitly add people to the dashboard group that is created if I create a new dashboard though - simply choosing "all users" doesn't seem to work afaict. It is a minor thing, so it's not stopping me from using it, but it would of course be nice if that could be fixed at some point.
    Dan Lester
    @AxelTLarsson Great to hear it works well so far! It was only a first attempt really. I can't reproduce the permissions problem though. The named per-dashboard group (e.g. dash-test) should only be relevant when 'Selected Users' is chosen. Group membership can be changed either on the dashboard edit page (unless All Users is chosen) or on the new Group page (for admins). But the group shouldn't have any bearing on access when All Users is chosen.
    It's possible that cookies are confused, especially if testing by logging in/out on the same browser. Please could you try again, and maybe spell out steps if you still think there is a fundamental problem (e.g. did user exist when dashboard was started?)
    But in any case I need to take a look at how JupyterHub handles cookies on logout
    Dan Lester
    BTW in a commit just now I moved Groups menu item into the top of the Admin page. So click Admin to be able to get to it.
    Axel Larsson
    Yeah, actually, I can't reproduce the issue now, and definitely the cookies could have some effect, because I tested with my own user, adding/removing it from the spawners-group. Makes sense to have the group functionality under Admin!
    Dan Lester
    Great, let me know any other problems and/or feedback etc.
    Sid Feygin
    @danlester Thanks for the rapid implementation on a group management ui! I see the commits you've added, but I'm unsure how to deploy this on k8s, since it's not yet tagged with a versioned release.
    Dan Lester
    Good question @sfwatergit. The jupyterhub image is built for every commit, for example cdsdashboards commit b5a2950 ends up on Docker hub tagged as ideonate/cdsdashboards-jupyter-k8s-hub:sha-b5a2950
    However, there may not be a corresponding singleuser image, but in this case the latest singleuser images should work fine anyway.
    It would be great if you can try out the new functionality, but of course it's experimental. Feedback is essential!
    Sid Feygin
    @danlester : I've been able to try out the groups management ui, but I'm not exactly sure what it is meant to accomplish. When I click on the manage_groups link, I get a list of all servers and can add or remove users from the servers.
    7 replies
    Victor Tolpegin

    Hey all, I'm really enjoying using dashboards. But I've recently run into an issue...When I start a dashboard, then stop the server, and finally restart the server, I sometimes get the following error:

    ERROR:tornado.application:Uncaught exception GET /user/****/dash-testsds/ (****)
    HTTPServerRequest(protocol='http', host='****:8888', method='GET', uri='/user/****/dash-testsds/', version='HTTP/1.1', remote_ip='****')
    Traceback (most recent call last):
      File "/opt/amazon/lib/python3.7/site-packages/tornado/web.py", line 1592, in _execute
        result = yield result
      File "/opt/amazon/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/usr/local/lib/python3.7/site-packages/jhsingle_native_proxy/websocket.py", line 103, in get
        return await self.http_get(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 718, in http_get
        return await self.proxy(self.port, path)
      File "/usr/local/lib/python3.7/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 712, in proxy
        return await self.oauth_proxy(port, path)
    TypeError: object NoneType can't be used in 'await' expression

    Any thoughts why this is happening?

    10 replies
    If I restart the server again (after seeing this error), it sometimes fixes itself and I can use it again
    Sid Feygin

    Hi! I've found that when I use rather memory-intensive widgets with voila, I get failed page loads (timeout from Tornado). I found the following error in my logs when this happens:

    ERROR:asyncio:Task exception was never retrieved
    future: <Task finished name='Task-33' coro=<SuperviseAndProxyHandler.ensure_process.<locals>.pipe_output() done, defined at /opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py:645> exception=ValueError('Separator is found, but chunk is longer than limit')>
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/asyncio/streams.py", line 540, in readline
        line = await self.readuntil(sep)
      File "/opt/conda/lib/python3.8/asyncio/streams.py", line 635, in readuntil
        raise exceptions.LimitOverrunError(
    asyncio.exceptions.LimitOverrunError: Separator is found, but chunk is longer than limit
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 650, in pipe_output
        line = await stream.readline()
      File "/opt/conda/lib/python3.8/asyncio/streams.py", line 549, in readline
        raise ValueError(e.args[0])
    ValueError: Separator is found, but chunk is longer than limit

    Any ideas on this?

    Dan Lester
    Thanks @sfwatergit - would it be possible to produce a minimal Voila script to simulate this so I can try it out? Or at least some idea of the operations that are in progress when this happens? If a GitHub issue or email is better to share this, please feel free!
    13 replies
    Are containers shared between users?
    1 reply
    Justin Angevaare
    Hello. I installed containds to tljh server - very easy, thank you. I've deployed a test dashboard fine (single file plotly dash that you include), but unable to deploy a dashboard that is broken up across a couple files. Any tips?
    Error report from ContainDS Dashboards
    Command Running:
    python3 -m plotlydash_tornado_cmd.main /home/jupyter-justina/./Network/main.py --port=49503
    Error output:
    Fetching Plotly Dash script /home/jupyter-justina/./Network/main.py
    CWD to /home/jupyter-justina/./Network
    Importing user Dash app
    Standard output:
    Traceback (most recent call last):
      File "/opt/tljh/user/lib/python3.7/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/opt/tljh/user/lib/python3.7/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/opt/tljh/user/lib/python3.7/site-packages/plotlydash_tornado_cmd/main.py", line 88, in <module>
      File "/opt/tljh/user/lib/python3.7/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/opt/tljh/user/lib/python3.7/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/opt/tljh/user/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/opt/tljh/user/lib/python3.7/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/opt/tljh/user/lib/python3.7/site-packages/plotlydash_tornado_cmd/main.py", line 75, in run
        app = make_app(command, server_name, debug)
      File "/opt/tljh/user/lib/python3.7/site-packages/plotlydash_tornado_cmd/main.py", line 35, in make_app
      File "<frozen importlib._bootstrap_external>", line 728, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/home/jupyter-justina/./Network/main.py", line 11, in <module>
        import layout
    ModuleNotFoundError: No module named 'layout'
    10 replies
    Justin Angevaare
    Is it possible/would it be possible to help with management of debug/development servers as well within containDS?
    9 replies
    Hi all, I have installed cdsdashboards on z2jh. The first time I create the dashboard, the dashboard server starts up just fine, but when I stop and restart the server its goes through the profilelist wizard, presents the server otpions that we have setup and then it starts up as regular server, spawner_class has been set to VariableKubeSpawner in the config. Any ideas?
    4 replies
    Ricky Lim

    Hi All, I have a question concerning users restriction with cdsdashboard.

    Is there a way to restrict the spawning scope, such that a user can only spawn her/his own jupyter server, but NOT for the named-server for the dashboard?

    Any pointer would be appreciated. Cheers

    1 reply
    Sid Feygin

    @danlester : Getting back to integration of ContainDS Dashboards this year. We've successfully embedded jupyterhub in an iframe; however, I am unable to get dashboards to display in the iframe due to its violating the Content Security Policy. To get around this issue in notebooks served by JupyterHub as well as JupyterHub itself, I've followed jupyterhub/jupyterhub#379, adding the following to my config.yaml:

     single-user: |
          c.JupyterHub.tornado_settings = { 'headers': { 'Content-Security-Policy': 'frame-ancestors self https://*.domain.name/'}}
        hub: |
          c.JupyterHub.tornado_settings = { 'headers': { 'Content-Security-Policy': 'frame-ancestors self https://*.domain.name/'}}
        spawner: |
          c.Spawner.args = ['--NotebookApp.tornado_settings={"headers":{"Content-Security-Policy": "frame-ancestors * self https://*.domain.name/"}}']

    (where, to be clear, in reality, domain.name substituted appropriately for our custom domain).
    However, CDashboards proxies voila via jhsingle-native-proxy, so we cannot pass --NotebookApp.tornado_setting to voila. An voila-dashboards/voila#609 on the Voila GH indicates that embedding Voila in an iframe should be able to succeed if I were to replace the last line of the above with the following instead:

    c.Spawner.args = ['{--}Voila.tornado_settings="{\'headers\':{\'Content-Security-Policy\': \"frame-ancestors * self https://*.domain.name/\"}}"']

    This is an improvement (of sorts), since it now passes the setting to tornado, but I end up with the following error:

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/websocket.py", line 103, in get
        return await self.http_get(*args, **kwargs)
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 724, in http_get
        return await self.proxy(self.port, path)
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 718, in proxy
        return await self.oauth_proxy(port, path)
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 673, in oauth_proxy
        return await self.core_proxy(port, path)
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 704, in core_proxy
        if not await self.ensure_process():
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 603, in ensure_process
        cmd = self.get_cmd()
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 840, in get_cmd
        return self._render_template(command)
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 827, in _render_template
        return [self._render_template(v) for v in value]
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 827, in <listcomp>
        return [self._render_template(v) for v in value]
      File "/opt/conda/lib/python3.8/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 825, in _render_template
        return value.format(**args)
    KeyError: "'headers'"

    Any idea what I can do to fix this?

    Dan Lester

    Great to hear from you!

    As I think you suggest, it would be ideal if jhsingle-native-proxy itself would accept the tornado_settings directly (so Voila doesn't even need to be involved). For one thing, in that case you would pass something more like ['--tornado_settings=... instead of it being a level deeper and need quite so much escaping.

    It might also be able to apply to more than just Voila in one go. By the way, you might also be interested in seeing how to modify the Voila 'launcher' directly here: https://cdsdashboards.readthedocs.io/en/stable/chapters/customization/customlaunchers.html

    (The same approach for adding your own presentation type can be used to modify one of the built-in ones.)

    It might still be possible to fix your problem directly. The problem is that jhsingle-native-proxy is attempting to substitute a fixed list of variables into the args string, namely:

                    'port': self.port,
                    'base_url': self.base_url,
                    'presentation_path': self.presentation_path,
                    'presentation_basename': self.presentation_basename,
                    'presentation_dirname': self.presentation_dirname,
                    'origin_host': self.origin_host,
                    '-': '-',
                    '--': '--'

    So for example, {port} is replaced with the value of port - this is just using python format function.

    However, it encounters {\'headers... which it thinks is starting a variable substituiton that doesn't exist.

    To fix this in Python format generally, you use a double brace, e.g. {{ to escape a curly brace to produce a literal { instead of starting a sub.

    I haven't tried it but maybe:

    c.Spawner.args = ['{--}Voila.tornado_settings="{{\'headers\':{{\'Content-Security-Policy\': \"frame-ancestors * self https://*.domain.name/\"}}}}"']

    but there might be more escaping needed in the ' and " or something yet!