Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Oct 15 20:58

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 15 20:57

    cgentemann on master

    Update meeting-notes.rst added… (compare)

  • Oct 14 10:40

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 14 10:39

    rabernat on master

    Update copyright year to 2018-2… (compare)

  • Oct 14 10:39
    rabernat closed #853
  • Oct 14 10:39
    rabernat commented #853
  • Oct 13 13:32

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 13 13:32

    leslie-yoc on master

    Update pangeo-showcase.rst Add… Merge pull request #859 from pa… (compare)

  • Oct 13 13:32
    leslie-yoc closed #859
  • Oct 13 13:31
    github-actions[bot] commented #859
  • Oct 13 13:31

    github-actions[bot] on leslie-yoc-patch-5-preview

    Update documentation (compare)

  • Oct 13 13:31
    leslie-yoc opened #859
  • Oct 13 13:30

    leslie-yoc on leslie-yoc-patch-5

    Update pangeo-showcase.rst Add… (compare)

  • Oct 11 17:31

    leslie-yoc on leslie-yoc-patch-4

    Update pangeo-showcase.rst Add… (compare)

  • Oct 11 16:21

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 11 16:20

    leslie-yoc on master

    Update pangeo-showcase.rst Upd… (compare)

  • Oct 06 15:20

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 06 15:19

    rabernat on master

    Update pangeo-showcase.rst (#85… (compare)

  • Oct 06 15:19
    rabernat closed #858
  • Oct 06 15:18

    github-actions[bot] on leslie-yoc-patch-3-preview

    Update documentation (compare)

Ian Rose
@ian-r-rose
Yes, nbserverproxy has been significantly updated and renamed to jupyter-server-proxy
David Hoese
@djhoese
ah I see hub-extension isn't actually listed there specifically (I realize it is a JS dependency)
ok so I'll try switching that and see what happens
David Hoese
@djhoese
thanks and I suppose --py nbserverproxy is wrong too, researching...
jupyter_server_proxy, got it
David Hoese
@djhoese
@ian-r-rose FYI I got an error about not being able to list dask clusters when jlab opened. I removed the dask panes from the workspace, but I'm guessing this is part of what @TomAugspurger was hoping to look at soon.
It still let's me use the interface it seems so not a big deal right now
Scott Henderson
@scottyhq
@philippjfr and @jhamman - just getting to this, looks like the cluster autoscaler went down last week kube-system cluster-autoscaler-8d5bc58f-4d76t 0/1 ImagePullBackOff 0 9d
Ian Rose
@ian-r-rose
@djhoese Can you open an issue? It would be especially helpful if you have a binder link I could access.
David Hoese
@djhoese
Will do
Ian Rose
@ian-r-rose
Thanks!
David Hoese
@djhoese
on the cookiecutter or where @ian-r-rose ?
Ian Rose
@ian-r-rose
dask-labextension would be great
Philipp Rudiger
@philippjfr
@scottyhq Thanks for looking into it.
Scott Henderson
@scottyhq
restarting the autoscaler didn’t fix it. in the meantime icesat2.pangeo.io seems to be working (running in us-west-2)
Philipp Rudiger
@philippjfr
I was hoping to use the NFS mounted home directory on staging.nasa.pangeo.io so I can modify the environment. Is that set up for icesat2?
Scott Henderson
@scottyhq
ah, right, no it isn’t. staging.esip.pangeo.io then ;)
i’m currently sitting w/ rich trying to update that one, so it might also be down from time to time
Philipp Rudiger
@philippjfr
Perfect, thanks! :)
Scott Henderson
@scottyhq
no problem, unfortunately won’t have time to fix nasa.pangeo.io until later today… on the move
Fabien Maussion
@fmaussion
Folks, a non important question out of curiosity: why are you using google docs for the weekly notes instead of HackMd? (http://pangeo.io/meeting-notes.html) - was it in order to be more inclusive? (hackmd is still niche)
Scott Henderson
@scottyhq
@philippjfr - nasa.pangeo.io is back in action
Philipp Rudiger
@philippjfr
@scottyhq Any chance it's gone down again?
Philipp Rudiger
@philippjfr
Also are you sure staging.esip.pangeo.io mounts the home directory? Doesn't seem to when I try.
Scott Henderson
@scottyhq
nasa.pangeo.io is working fine for me @philippjfr . are you running into errors? you’re right the esip config doesn’t mount the same home directory for dask workers.
Philipp Rudiger
@philippjfr
I'll try again now.
Philipp Rudiger
@philippjfr
If nasa.pangeo.io works should staging.nasa.pangeo.io work too?
Yay, I can confirm, it's working for me too now.
Scott Henderson
@scottyhq
yep. at least the way things are currently set up, there is a single cluster autoscaler that manages both staging and prod
Philipp Rudiger
@philippjfr
Cool, it's still not picking up on my custom conda env unfortunately. This should work right?
cluster = KubeCluster(env={'PATH': '/home/jovyan/my-conda-envs/datashader_dev/bin:$PATH'}, n_workers=2)
Scott Henderson
@scottyhq
hmm, so that used to work! but i just checked and it no longer does the trick. maybe due to changes to the way repo2docker changed the conda environment setup (pangeo-data/pangeo-stacks#47)
i thought maybe adding ‘CONDA_PREFIX’ or ‘CONDA_DEFAULT_ENV’ but at some point when workers are initialized, the ‘notebook’ environment keeps ending up in front : 'PATH': '/srv/conda/envs/notebook/bin:/srv/conda/condabin:/home/jovyan/my-conda-envs/dask-minimal/bin:/srv/conda/condabin:/home/jovyan/.local/bin:/home/jovyan/.local/bin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin’
you can check all the environment variables on the workers with client.run(lambda: os.environ)
Philipp Rudiger
@philippjfr
Ah nice, didn't know about client.run, used the much more hacky delayed(fn)().compute().
Scott Henderson
@scottyhq
@philippjfr - this seems to work!
cluster = KubeCluster(env={'NB_PYTHON_PREFIX':sys.prefix})
Philipp Rudiger
@philippjfr
Cool! I was hacking the sys.path after the fact, but that seems a bit nicer.
One more thing, does anyone have a really large zarr dataset I could test distributed regridding code on?
Joe Hamman
@jhamman
Most (maybe all) of these datasets are in GCP central though.
Scott Henderson
@scottyhq
we can move any of those you might be interested in experimenting with to s3://pangeo-data-useast1 that is accessible from nasa.pangeo.io (there are only a few datasets in that bucket currently)
Philipp Rudiger
@philippjfr
That would be really great. Out of those the Hydro cgiar_pet seems perfect. The hydrosheds seem interesting too but I'm not quite clear on how they are structured, is there a time dimension that's not made explicit?
David Brochart
@davidbrochart
@philippjfr The hydrosheds datasets don't have any time dimension. They are GDAL VRTs, you can open them with e.g.:
import xarray as xr
import gcsfs

fs = gcsfs.GCSFileSystem('pangeo-data')
fs.get('pangeo-data/hydrosheds/acc.vrt', './acc.vrt')
da = xr.open_rasterio('./acc.vrt')
(da.sel(band=1, x=slice(-60, -59), y=slice(1, 0)) ** 0.1).plot()
Tina Odaka
@tinaok
@kmpaul @guillaumeeb @willirath , I'm just back from holiday, trying to catch up the situation. Does anyone have any updates on Pangeo participation on SC19? I think it would be a great occasion to talk about HPC integration of pangeo, optimisation, also show our test cases and benchmarks.
Kevin Paul
@kmpaul
@tinaok Yes! SC19 is in my backyard, this year, so we will be there. There is a workshop on interactive HPC that I think a lot of Pangeo would fit into well. However, there are other workshops that would be good fits for other topics such as (maybe) a scientific data reduction workshop, a workshop on cloud-HPC interoperability and maybe others. I think the interactive HPC workshop would be a good venue, though.
@tinaok Actually, I’ve been meaning to connect with you on this for a while, but I’ve been busy and forgot. Thanks for reaching out!
Tina Odaka
@tinaok
@kmpaul Wonderfull, I am looking for someone who I can make joint presentation on usage of pangeo (my domain is more like benchmarking, optimal usage of HPC, but I can expand.) on workshop of BoF. Interactive HPC sounds good too.
Kevin Paul
@kmpaul
I’d be happy to join you in that, @tinaok. Count me in!