Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Oct 15 20:58

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 15 20:57

    cgentemann on master

    Update meeting-notes.rst added… (compare)

  • Oct 14 10:40

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 14 10:39

    rabernat on master

    Update copyright year to 2018-2… (compare)

  • Oct 14 10:39
    rabernat closed #853
  • Oct 14 10:39
    rabernat commented #853
  • Oct 13 13:32

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 13 13:32

    leslie-yoc on master

    Update pangeo-showcase.rst Add… Merge pull request #859 from pa… (compare)

  • Oct 13 13:32
    leslie-yoc closed #859
  • Oct 13 13:31
    github-actions[bot] commented #859
  • Oct 13 13:31

    github-actions[bot] on leslie-yoc-patch-5-preview

    Update documentation (compare)

  • Oct 13 13:31
    leslie-yoc opened #859
  • Oct 13 13:30

    leslie-yoc on leslie-yoc-patch-5

    Update pangeo-showcase.rst Add… (compare)

  • Oct 11 17:31

    leslie-yoc on leslie-yoc-patch-4

    Update pangeo-showcase.rst Add… (compare)

  • Oct 11 16:21

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 11 16:20

    leslie-yoc on master

    Update pangeo-showcase.rst Upd… (compare)

  • Oct 06 15:20

    github-actions[bot] on gh-pages

    Update documentation (compare)

  • Oct 06 15:19

    rabernat on master

    Update pangeo-showcase.rst (#85… (compare)

  • Oct 06 15:19
    rabernat closed #858
  • Oct 06 15:18

    github-actions[bot] on leslie-yoc-patch-3-preview

    Update documentation (compare)

David Hoese
@djhoese
on the cookiecutter or where @ian-r-rose ?
Ian Rose
@ian-r-rose
dask-labextension would be great
Philipp Rudiger
@philippjfr
@scottyhq Thanks for looking into it.
Scott Henderson
@scottyhq
restarting the autoscaler didn’t fix it. in the meantime icesat2.pangeo.io seems to be working (running in us-west-2)
Philipp Rudiger
@philippjfr
I was hoping to use the NFS mounted home directory on staging.nasa.pangeo.io so I can modify the environment. Is that set up for icesat2?
Scott Henderson
@scottyhq
ah, right, no it isn’t. staging.esip.pangeo.io then ;)
i’m currently sitting w/ rich trying to update that one, so it might also be down from time to time
Philipp Rudiger
@philippjfr
Perfect, thanks! :)
Scott Henderson
@scottyhq
no problem, unfortunately won’t have time to fix nasa.pangeo.io until later today… on the move
Fabien Maussion
@fmaussion
Folks, a non important question out of curiosity: why are you using google docs for the weekly notes instead of HackMd? (http://pangeo.io/meeting-notes.html) - was it in order to be more inclusive? (hackmd is still niche)
Scott Henderson
@scottyhq
@philippjfr - nasa.pangeo.io is back in action
Philipp Rudiger
@philippjfr
@scottyhq Any chance it's gone down again?
Philipp Rudiger
@philippjfr
Also are you sure staging.esip.pangeo.io mounts the home directory? Doesn't seem to when I try.
Scott Henderson
@scottyhq
nasa.pangeo.io is working fine for me @philippjfr . are you running into errors? you’re right the esip config doesn’t mount the same home directory for dask workers.
Philipp Rudiger
@philippjfr
I'll try again now.
Philipp Rudiger
@philippjfr
If nasa.pangeo.io works should staging.nasa.pangeo.io work too?
Yay, I can confirm, it's working for me too now.
Scott Henderson
@scottyhq
yep. at least the way things are currently set up, there is a single cluster autoscaler that manages both staging and prod
Philipp Rudiger
@philippjfr
Cool, it's still not picking up on my custom conda env unfortunately. This should work right?
cluster = KubeCluster(env={'PATH': '/home/jovyan/my-conda-envs/datashader_dev/bin:$PATH'}, n_workers=2)
Scott Henderson
@scottyhq
hmm, so that used to work! but i just checked and it no longer does the trick. maybe due to changes to the way repo2docker changed the conda environment setup (pangeo-data/pangeo-stacks#47)
i thought maybe adding ‘CONDA_PREFIX’ or ‘CONDA_DEFAULT_ENV’ but at some point when workers are initialized, the ‘notebook’ environment keeps ending up in front : 'PATH': '/srv/conda/envs/notebook/bin:/srv/conda/condabin:/home/jovyan/my-conda-envs/dask-minimal/bin:/srv/conda/condabin:/home/jovyan/.local/bin:/home/jovyan/.local/bin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin’
you can check all the environment variables on the workers with client.run(lambda: os.environ)
Philipp Rudiger
@philippjfr
Ah nice, didn't know about client.run, used the much more hacky delayed(fn)().compute().
Scott Henderson
@scottyhq
@philippjfr - this seems to work!
cluster = KubeCluster(env={'NB_PYTHON_PREFIX':sys.prefix})
Philipp Rudiger
@philippjfr
Cool! I was hacking the sys.path after the fact, but that seems a bit nicer.
One more thing, does anyone have a really large zarr dataset I could test distributed regridding code on?
Joe Hamman
@jhamman
Most (maybe all) of these datasets are in GCP central though.
Scott Henderson
@scottyhq
we can move any of those you might be interested in experimenting with to s3://pangeo-data-useast1 that is accessible from nasa.pangeo.io (there are only a few datasets in that bucket currently)
Philipp Rudiger
@philippjfr
That would be really great. Out of those the Hydro cgiar_pet seems perfect. The hydrosheds seem interesting too but I'm not quite clear on how they are structured, is there a time dimension that's not made explicit?
David Brochart
@davidbrochart
@philippjfr The hydrosheds datasets don't have any time dimension. They are GDAL VRTs, you can open them with e.g.:
import xarray as xr
import gcsfs

fs = gcsfs.GCSFileSystem('pangeo-data')
fs.get('pangeo-data/hydrosheds/acc.vrt', './acc.vrt')
da = xr.open_rasterio('./acc.vrt')
(da.sel(band=1, x=slice(-60, -59), y=slice(1, 0)) ** 0.1).plot()
Tina Odaka
@tinaok
@kmpaul @guillaumeeb @willirath , I'm just back from holiday, trying to catch up the situation. Does anyone have any updates on Pangeo participation on SC19? I think it would be a great occasion to talk about HPC integration of pangeo, optimisation, also show our test cases and benchmarks.
Kevin Paul
@kmpaul
@tinaok Yes! SC19 is in my backyard, this year, so we will be there. There is a workshop on interactive HPC that I think a lot of Pangeo would fit into well. However, there are other workshops that would be good fits for other topics such as (maybe) a scientific data reduction workshop, a workshop on cloud-HPC interoperability and maybe others. I think the interactive HPC workshop would be a good venue, though.
@tinaok Actually, I’ve been meaning to connect with you on this for a while, but I’ve been busy and forgot. Thanks for reaching out!
Tina Odaka
@tinaok
@kmpaul Wonderfull, I am looking for someone who I can make joint presentation on usage of pangeo (my domain is more like benchmarking, optimal usage of HPC, but I can expand.) on workshop of BoF. Interactive HPC sounds good too.
Kevin Paul
@kmpaul
I’d be happy to join you in that, @tinaok. Count me in!
Scott Henderson
@scottyhq
@davidbrochart and @philippjfr i’ve gone ahead and copied hydrosheds over to s3, so you can now access from either hub:
import xarray as xr
import s3fs
fs = s3fs.S3FileSystem(anon=False, requester_pays=True)
fs.get('pangeo-data-useast1/hydrosheds/acc.vrt', './acc.vrt')
da = xr.open_rasterio('./acc.vrt')
da
Philipp Rudiger
@philippjfr
@scottyhq Great, does that include the cgiar_pet dataset?

Also my server seems to keep crashing on nasa.pangeo.io:

Server Connection Error
Invalid response: 503

Scott Henderson
@scottyhq
hmm.. i’m seeing Evicted Pods on the cluster, no idea what the cause might be. let me know next time you encounter the Error and I might be able to glean more
i’ll transfer cgiar_pet as well
Joe Hamman
@jhamman
@scottyhq - I just got a connection error too!
Scott Henderson
@scottyhq
hmm... message: 'The node was low on resource: ephemeral-storage. Container notebook was using 28Ki, which exceeds its request of 0. ' phase: Failed reason: Evicted
Joe Hamman
@jhamman
hmmm, for your info, I hadn’t done anything in my session yet. It just created then died almost immediately.
Scott Henderson
@scottyhq
not exactly sure what fills it up, but looks like all the nodes are still launching w/ default 20Gb EBS disk (new clusters we updated to 100Gb)
Scott Henderson
@scottyhq
in fact we documented this earlier! pangeo-data/pangeo-cloud-federation#274
Scott Henderson
@scottyhq
i’ll have a bit of time this afternoon to make sure its updated
David Hoese
@djhoese

@jhamman At scipy you talked about how pangeo's binder can scale down to 0. Any idea how long it keeps things alive before shutting everything down? What about time to start up? What about timeout on individual inactive JLab sessions?

I was thinking if I load a repository X minutes before a tutorial then everyone should have speedy access to their JLab session. However, if I do it too early then we'll have to wait anyway.