KERNEL_LAUNCH_TIMEOUTconfig? I'm checking
jupyter-lab --help-allbut only see these two options available:
--GatewayClient.connect_timeout=<Float> Default: 40.0 The time allowed for HTTP connection establishment with the Gateway server. (JUPYTER_GATEWAY_CONNECT_TIMEOUT env var) --GatewayClient.request_timeout=<Float> Default: 40.0 The time allowed for HTTP request completion. (JUPYTER_GATEWAY_REQUEST_TIMEOUT env var)
we spoke in the past about the issues with having all requests funnel through a single proxy. there seems to be a pattern of adding components to the jupyter stack that mostly act as reverse proxies to components further down (hub → lab → eg → kernel) which adds a huge amount of stability issues. hub sitting at the very top of this funnel made it responsible for a lot of issues. having every cell execution run through the hub gave us latency issues and a crash of the hub instance generally meant everyone lost their lab instances, which caused people to lose all their kernels. deploying a change to the hub is a huge hassle for this same reason.
ultimately, I don't want something monitoring my lab instances. kubernetes has native functionality to represent services, monitor them, and ensure they remain healthy. I don't need a custom solution for jupyter pods that differs from the other services running in my cluster.
on top of that, we have a central auth solution that covers our entire cluster and any service running inside it. which means we have little use for the hub multi-user features. I don't need to handle authentication in jupyter
time.sleep(120)during startup to simulate slow launches and am noticing that if I launch two kernels, EG is not beginning to provision the second one until the first has fully started. The timeout setting has improved the issue though and eventually both kernels do run. am I missing some settings to make EG handle more than one request at a time?
@coder-forever-2020 - where are you using
jupyter/minimal-notebook:latest? Is this for your client-side notebook server? Please include the full traceback rather than just the "was never awaited" message.
Regarding additional envs, besides unconditional
KERNEL_-prefixed envs, you can add a list of env names to
EG_ENV_WHITELIST or from the CLI
--EnterpriseGatewayApp.env_whitelist which can also be included:
--EnterpriseGatewayApp.env_whitelist=<List> Default:  Environment variables allowed to be set when a client requests a new kernel. (EG_ENV_WHITELIST env var)
KERNEL_env variables, then modify the kernelspec files to use those values accordingly.
KERNEL_NUM_GPUS=2) into the
env:stanza of the kernel start request body. If you don't have control over the UI, then those envs would need to be present in the notebook process making the kernel start request since the gateway logic automatically flows any
I noticed the latest version of JupyterLab (3.0.0-rc.8) doesn't have the check_origin() that was added to the gateway websocket handler (https://github.com/jupyter/jupyter_server/blob/master/jupyter_server/gateway/handlers.py#L35).
In general, does Enterprise Gateway already support JupyterLab 3.0.0? Because even with the above patched, I couldn't get kernel messages flowing from JupyterLab (3.0.0-rc.8) to EG (tested with 2.2). No errors, just stuck (kernel had lightning icon) executing a cell.
Seems like same issue happens on 2.3 + 3.0.0-rc9. Created a PR with jupyter/jupyter_server for handler.py and created an issue for tracking interop on Enterprise Gateway.
Could be related to awaiting connect in the handler (see issue #903).
Thanks Rick. Yes, the check_origin change is good. However, I'm still unable to complete the kernel's startup from lab3 to EG. I'm not seeing a kernel-info-request/reply sequence that typically occurs during the establishment of the websocket - which I'm not seeing either.
There shouldn't be anything required on the EG side of things, but I need to ensure there aren't any missing PRs in notebook that belong in jupyter-server - like the check_origin change.
I love the idea of Enterprise Gateway and would love to take it for a spin. I am having some installation issues. Is there any way of testing this locally in KIND? (Kubernetes in docker?). I am running into issues with the
kernel-image-puller not being able to connect. Error 111.
│ Traceback (most recent call last): │ │ File "./kernel_image_puller.py", line 25, in <module> │ │ docker_client = DockerClient.from_env() │ │ File "/usr/local/lib/python3.8/site-packages/docker/client.py", line 84, in f │ │ rom_env │ │ return cls( │ │ File "/usr/local/lib/python3.8/site-packages/docker/client.py", line 40, in _ │ │ _init__ │ │ self.api = APIClient(*args, **kwargs) │ │ File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 188, │ │ in __init__ │ │ self._version = self._retrieve_server_version() │ │ File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 212, │ │ in _retrieve_server_version │ │ raise DockerException( │ │ docker.errors.DockerException: Error while fetching server API version: ('Conne │ │ ction aborted.', ConnectionRefusedError(111, 'Connection refused'))
/usr/local/share/jupyter/kernelsdirectories. I'd just pick a couple kernelspecs you're interested in and pull those images.