pangeo-data/Lobby

general Pangeo discussion

Scott

@scollis

And @TomAugspurger I am running the examples in the pangeo-example-notebook repo

Tested now on Chrome and Safari

Jim Crist-Harif

@jcrist

@scollis , I've seen this behavior with tornado 6 and jupyter-server-proxy (the proxy used to proxy the dashboard through the user's notebook session). In the user's notebook environment, you need tornado 5 installed.

jupyterhub/jupyter-server-proxy#109

Scott

@scollis

Thanks.. will play with the environment.yml file

Joe Hamman

@jhamman

@rabernat, @rsignell-usgs, @tjcrone - can I merge this: pangeo-data/pangeo-cloud-federation#362

Scott Henderson

@scottyhq

@jhamman - just left a comment

Scott Henderson

@scottyhq

@fmaussion - we aren't updating images on any fixed timeline. But separating ‘staging’ and ‘production’ hubs has been useful to test things out and minimize disruption (see https://github.com/pangeo-data/pangeo-cloud-federation). As you already noted, the user home directory is unaffected. We’ve hit a lot of snags over the last months w/ version compatibilty when trying to have the latest packages (e.g. jupyterlab1, dask + dask-labextension). see https://github.com/pangeo-data/pangeo-stacks.

Joe Hamman

@jhamman

@mrocklin - if you’re around, let’s get you sorted on Cheyenne.

Scott

@scollis

Hey @scottyhq Thanks for the idea of using the base docker container.. I will give that a go…. Still lots to learn for me!

Scott

@scollis

All this stuff is very dfferent to your quickstarts.. if you want, once the dust settles from my learning I can put some PRs together with some new quickstarts including updating the cookie cutter

Scott Henderson

@scottyhq

no problem @scollis! thanks for trying things out. PRs are more than welcome. especially w/ documentation. it’s a bit of a moving target right now as we’re constantly experimenting with things so it's hard to keep current.

Scott

@scollis

so the base docker.. does that have the dask and jupyterlab set ups? so no need to include that anymore in the binder dir environment.yml

ie the environment.yml should now only have domain specific packages?

Scott Henderson

@scottyhq

that’s the idea yes. because we’re using repo2docker you end up layering packages that are defined in the following places in order 1) https://github.com/jupyter/repo2docker/blob/master/repo2docker/buildpacks/conda/environment.yml 2) https://github.com/pangeo-data/pangeo-stacks/blob/master/base-notebook/binder/environment.yml and 3) your own environment.yml.

if you want different versions of these base packages, including those versions in your environment.yml should overwrite them

Scott

@scollis

Great

Scott Henderson

@scottyhq

https://github.com/scottyhq/pangeo-binder-test/tree/esip-tech-dive is a working current example (although I don’t automatically run the notebook to check for errors)

in the binder folder this line in Dockerfile is key FROM pangeo/base-notebook-onbuild:2019.08.08 . It fetches the dated image form https://github.com/pangeo-data/pangeo-stacks and then runs your environment.yml, postBuild, etc on top

Joe Hamman

@jhamman

@scottyhq - I think we’re good with pangeo-data/pangeo-cloud-federation#362. Please take a look around staging.hub.pangeo.io to see if you spot anything out of place.

Scott

@scollis

Thanks @scottyhq .. trying now. This is very nice

Scott

@scollis

@scottyhq assume it is the same for postBuild and other stuff in /binder/*

Scott Henderson

@scottyhq

that’s right. except unfortunately I don’t think we enabled the ‘start’ script.

Scott

@scollis

@scottyhq up and running nicely.. some issues with the dask integration into the jupyterlab panel space but stuff I can debug and iron our..

Radar Pangeo here we come!

Scott Henderson

@scottyhq

awesome! great news

Joe Hamman

@jhamman

@scottyhq - ideas on this error/warning from conda:

The following packages are causing the inconsistency:

  - conda-forge/linux-64::notebook==6.0.0=py37_0
  - conda-forge/linux-64::ipyleaflet==0.11.1=py37_1
  - conda-forge/noarch::geoviews==1.6.2=py_0
  - conda-forge/linux-64::widgetsnbextension==3.5.1=py37_0
  - conda-forge/noarch::ipywidgets==7.5.1=py_0
  - conda-forge/noarch::jupyter==1.0.0=py_2
  - conda-forge/noarch::nbconvert==5.5.0=py_0
  - conda-forge/noarch::hvplot==0.4.0=py_1
  - conda-forge/noarch::geoviews-core==1.6.2=py_0
  - conda-forge/noarch::nbgitpuller==0.7.0=py_0
  - conda-forge/linux-64::jupyterlab==1.0.4=py37_0
  - conda-forge/noarch::jupyter-server-proxy==1.1.0=py_0
  - conda-forge/noarch::jupyterlab_server==1.0.0=py_1

full details here: https://circleci.com/gh/pangeo-data/pangeo-cloud-federation/705

Matthew Rocklin

@mrocklin

@jhamman ah, sorry, I was on a flight

I'm also currently away from a full-sized USB port, maybe tomorrow though?

Joe Hamman

@jhamman

Yeah, sounds good. You’ll find me here tomorrow as well. :airplane:

Pier

@PhenoloBoy

@jhamman is it you that I've to contact to get the git-crypt symmetric key as described here? https://github.com/pangeo-data/pangeo-cloud-federation

Scott

@scollis

@jhamman is there a publication I can cite for Pangeo?

Joe Hamman

@jhamman

@scollis - nothing comprehensive just yet. However, we did publish the NSF proposal (https://figshare.com/articles/Pangeo_NSF_Earthcube_Proposal/5361094) and @guillaumeeb had this conference paper published:

Eynard-Bontemps, G., R Abernathey, J. Hamman, A. Ponte, W. Rath, 2019: The Pangeo Big Data Ecosystem and its use at CNES. In P. Soille, S. Loekken, and S. Albani, Proc. of the 2019 conference on Big Data from Space (BiDS’2019), 49-52. EUR 29660 EN, Publications Office of the European Union, Luxembourg. ISBN: 978-92-76-00034-1, doi:10.2760/848593.

@PhenoloBoy - we’re trying not to share the git-crypt key unless absolutely necessary. Can I ask what you are up to?

Scott

@scollis

Perfect. Thanks… This would be perfect for something like AMS BAMS or suchlike or even a glossy like Science..

Joe Hamman

@jhamman

We’re…trying...

We actually have something in review with Nature but things seem to have stalled out in a big way.

This is on my list of things to deal with today.

Matthew Rocklin

@mrocklin

@jhamman I'm around. I've submitted a job and am going to wait a while to see if it clears

Joe Hamman

@jhamman

cheyenne is down right now so you’ll need to be on casper.

Matthew Rocklin

@mrocklin

Well that's good to know :)

Joe Hamman

@jhamman

do we have a working slurm cluster right now?

https://jupyterhub.ucar.edu/dav will put you on casper (which uses slurm)

Matthew Rocklin

@mrocklin

I'm ssh'ing in

I've just discovered that we use SLURM rather than PBS

sbatch: error: You must specify an account (--account)

In [4]: print(cluster.job_script())
#!/usr/bin/env bash

#SBATCH -J dask-worker
#SBATCH -n 1
#SBATCH --cpus-per-task=16
#SBATCH --mem=60G
#SBATCH -t 00:30:00
#SBATCH -C skylake
JOB_ID=${SLURM_JOB_ID%;*}



/glade/u/home/mrocklin/miniconda/envs/dev/bin/python -m distributed.cli.dask_worker tcp://10.12.203.5:39794 --nthreads 16 --memory-limit 64.00GB --name dask-worker--${JOB_ID}-- --death-timeout 60 --interface ib0


In [5]: import dask

In [6]: dask.config.get("jobqueue.slurm")
Out[6]:
{'name': 'dask-worker',
 'cores': 1,
 'memory': '25 GB',
 'processes': 1,
 'interface': 'ib0',
 'walltime': '00:30:00',
 'job-extra': {'-C skylake': None},
 'death-timeout': 60,
 'local-directory': None,
 'shebang': '#!/usr/bin/env bash',
 'queue': None,
 'project': None,
 'extra': ['--interface', 'ib0'],
 'env-extra': [],
 'job-cpu': None,
 'job-mem': None,
 'log-directory': None}

Matthew Rocklin

@mrocklin

I'm good

Where communities thrive

People

Repo info

Activity