Tested now on Chrome and Safari
Screenshot https://imgur.com/KfQywNo
@fmaussion - we aren't updating images on any fixed timeline. But separating ‘staging’ and ‘production’ hubs has been useful to test things out and minimize disruption (see https://github.com/pangeo-data/pangeo-cloud-federation). As you already noted, the user home directory is unaffected. We’ve hit a lot of snags over the last months w/ version compatibilty when trying to have the latest packages (e.g. jupyterlab1, dask + dask-labextension). see https://github.com/pangeo-data/pangeo-stacks.
ie the environment.yml should now only have domain specific packages?
that’s the idea yes. because we’re using repo2docker you end up layering packages that are defined in the following places in order 1) https://github.com/jupyter/repo2docker/blob/master/repo2docker/buildpacks/conda/environment.yml 2) https://github.com/pangeo-data/pangeo-stacks/blob/master/base-notebook/binder/environment.yml and 3) your own environment.yml.
if you want different versions of these base packages, including those versions in your environment.yml should overwrite them
https://github.com/scottyhq/pangeo-binder-test/tree/esip-tech-dive is a working current example (although I don’t automatically run the notebook to check for errors)
in the binder folder this line in Dockerfile is key
FROM pangeo/base-notebook-onbuild:2019.08.08 . It fetches the dated image form https://github.com/pangeo-data/pangeo-stacks and then runs your environment.yml, postBuild, etc on top
@scottyhq - I think we’re good with pangeo-data/pangeo-cloud-federation#362. Please take a look around staging.hub.pangeo.io to see if you spot anything out of place.
Radar Pangeo here we come!
@scottyhq - ideas on this error/warning from conda:
The following packages are causing the inconsistency:
- conda-forge/linux-64::notebook==6.0.0=py37_0
- conda-forge/linux-64::ipyleaflet==0.11.1=py37_1
- conda-forge/noarch::geoviews==1.6.2=py_0
- conda-forge/linux-64::widgetsnbextension==3.5.1=py37_0
- conda-forge/noarch::ipywidgets==7.5.1=py_0
- conda-forge/noarch::jupyter==1.0.0=py_2
- conda-forge/noarch::nbconvert==5.5.0=py_0
- conda-forge/noarch::hvplot==0.4.0=py_1
- conda-forge/noarch::geoviews-core==1.6.2=py_0
- conda-forge/noarch::nbgitpuller==0.7.0=py_0
- conda-forge/linux-64::jupyterlab==1.0.4=py37_0
- conda-forge/noarch::jupyter-server-proxy==1.1.0=py_0
- conda-forge/noarch::jupyterlab_server==1.0.0=py_1
full details here: https://circleci.com/gh/pangeo-data/pangeo-cloud-federation/705
I'm also currently away from a full-sized USB port, maybe tomorrow though?
@jhamman is it you that I've to contact to get the git-crypt symmetric key as described here? https://github.com/pangeo-data/pangeo-cloud-federation
@scollis - nothing comprehensive just yet. However, we did publish the NSF proposal (https://figshare.com/articles/Pangeo_NSF_Earthcube_Proposal/5361094) and @guillaumeeb had this conference paper published:
Eynard-Bontemps, G., R Abernathey, J. Hamman, A. Ponte, W. Rath, 2019: The Pangeo Big Data Ecosystem and its use at CNES. In P. Soille, S. Loekken, and S. Albani, Proc. of the 2019 conference on Big Data from Space (BiDS’2019), 49-52. EUR 29660 EN, Publications Office of the European Union, Luxembourg. ISBN: 978-92-76-00034-1, doi:10.2760/848593.
@PhenoloBoy - we’re trying not to share the git-crypt key unless absolutely necessary. Can I ask what you are up to?
We actually have something in review with Nature but things seem to have stalled out in a big way.
This is on my list of things to deal with today.
https://jupyterhub.ucar.edu/dav will put you on casper (which uses slurm)
I've just discovered that we use SLURM rather than PBS
sbatch: error: You must specify an account (--account)In [4]: print(cluster.job_script())
#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -n 1
#SBATCH --cpus-per-task=16
#SBATCH --mem=60G
#SBATCH -t 00:30:00
#SBATCH -C skylake
JOB_ID=${SLURM_JOB_ID%;*}
/glade/u/home/mrocklin/miniconda/envs/dev/bin/python -m distributed.cli.dask_worker tcp://10.12.203.5:39794 --nthreads 16 --memory-limit 64.00GB --name dask-worker--${JOB_ID}-- --death-timeout 60 --interface ib0
In [5]: import dask
In [6]: dask.config.get("jobqueue.slurm")
Out[6]:
{'name': 'dask-worker',
'cores': 1,
'memory': '25 GB',
'processes': 1,
'interface': 'ib0',
'walltime': '00:30:00',
'job-extra': {'-C skylake': None},
'death-timeout': 60,
'local-directory': None,
'shebang': '#!/usr/bin/env bash',
'queue': None,
'project': None,
'extra': ['--interface', 'ib0'],
'env-extra': [],
'job-cpu': None,
'job-mem': None,
'log-directory': None}