by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jul 03 20:29
    jhamman closed #775
  • Jul 03 20:29
    jhamman commented #775
  • Jul 03 18:55
    stale[bot] labeled #772
  • Jul 03 18:54
    stale[bot] labeled #775
  • Jul 03 18:54
    stale[bot] commented #772
  • Jul 03 18:54
    stale[bot] commented #775
  • Jul 03 06:15
    Boes-man commented #773
  • Jul 02 13:00
    consideRatio commented #773
  • Jul 02 12:22
    consideRatio commented #773
  • Jul 02 12:10
    Boes-man commented #773
  • Jul 02 11:57
    Boes-man commented #773
  • Jul 02 03:29

    rabernat on gh-pages

    Update docs after building Trav… (compare)

  • Jul 02 03:27

    rabernat on master

    Mega update of website content … (compare)

  • Jul 02 03:27
    rabernat closed #780
  • Jul 02 03:20
    rabernat synchronize #780
  • Jul 01 19:59
    rabernat synchronize #780
  • Jul 01 18:38
    rabernat synchronize #780
  • Jun 30 15:29
    rabernat commented #773
  • Jun 30 14:44
    guillaumeeb commented #773
  • Jun 30 11:42
    rabernat commented #773
James A. Bednar
@jbednar
@rsignell-usgs would probably know...
Dr. Andreas Hopfgartner
@anderl80
Hi all, I have a Pangeo Cluster in my gcloud. I'm not so familiar with gcloud (more Azure), is there a possibility to shut down the cluster for having no costs?
Matthew Rocklin
@mrocklin
Dask Jobqueue + SpecCluster rewrite issue here, if anyone wants to play around with HPC systems: dask/dask-jobqueue#306
Ryan May
@dopplershift
@djhoese The work going in netcdf-c will make zarr work as 'just' another on-disk format for the library to use, like it can use HDF5 or netCDF3. Really, though, this is a great question to send to the netCDF user list: https://www.unidata.ucar.edu/software/netcdf/mailing-lists.html (or support-netcdf@unidata.ucar.edu)
@djhoese As far as the current data in S3 (and maybe GCS), there is support in the library now to make byte-range requests over HTTP, so I think you could do a direct read of hdf5-backed data in S3 using the library.
David Hoese
@djhoese
@dopplershift any idea how it encodes that in the HTTP request? Doesn't that have to be understood by the server? I guess that is important enough that it was probably put in a while ago
Ryan May
@dopplershift
Byte ranges are a standard http request header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
Tom Augspurger
@TomAugspurger

Is anyone here using ocean.pangeo.io heavily right now? pangeo-data/pangeo-cloud-federation#360. Trying to figure out why it’s giving a 504.

It looks like there’s a bunch of dask worker pods running. Trying to figure out the hub isn’t handling new users though.

Joe Hamman
@jhamman
hey @TomAugspurger. I’m here now
Eduardo Gonzalez
@eddienko

I have deployed pango in my local cluster (thanks for making it easy!). I do have question about the dask integration. I have a few volumes containing data that are mounted in the jupyter notebook server and defined in the cluster specific configuration jupyter_config.yaml (using extraVolumes and extraVolumeMounts). So far so good.

The problem is that when launching dask using KubeCluster from the notebook, these volumes are not available from the dask workers. Is there an 'automatic' way of doing this or do I need to define the volumes as well in the dask.yaml file? Or am I gettting this wrong (possibly!)?

Joe Hamman
@jhamman
Glad to hear it’s going well so far. Yes, you need to mount the volumes in your workers as well.
Eduardo Gonzalez
@eddienko
Ok, thanks. EXTRA_PIP_PACKAGES is not propagated neither. I wrote this in case it is useful: https://gist.github.com/eddienko/7b5b40ff18309ecd7f28d294c3f1cd72
Joe Hamman
@jhamman
I think the EXTRA_PIP_PACKAGES option is no more. Where did you find that?
Eduardo Gonzalez
@eddienko
This is linked from the official Pangeo.io documentation: https://github.com/pangeo-data/pangeo/blob/master/gce/setup-guide/jupyter_config.yaml
Ryan Abernathey
@rabernat
How can we get diagnostics on our google cloud firestore?
like disk usage, etc.
Joe Hamman
@jhamman
No idea but this seems useful.
Ryan Abernathey
@rabernat
I'm worried that ocean hack week users are going to consume a lot of home storage space on ocean.pangeo.io
it would be good to have some monitoring
Scott
@scollis
Hey Folks.. Thanks to some help from @jhamman I am up and running. I was running into an issue that I did not get the nice views on the panels when I was running Dask.. (profiler et al).. So I ran the example Pangeo notebooks using the binder link. http://binder.pangeo.io/v2/gh/pangeo-data/pangeo-example-notebooks/master and I get the same issue that I never get nice output..even when I click the dask icon in the right bar I can not see what workers are doing.. Could there be a network blocking issue (some stuff is blocked here at Argonne)?
is blank
Wonder if it is a Bokeh issue...
Fabien Maussion
@fmaussion
Hi Pangeo folks: how often do you actualize the base image on hub.pangeo.io (or do you even do that?)
What are the consequences for the users? I expect these to be minimal, i.e. a user's HOME is completely decoupled from the base image
but still, what real bad could happen?
Scott
@scollis
So I tried on my home computer but I still get blank panels.. Just curious to see if this on only me (browser: Chrome, OS: MacOS)
Tom Augspurger
@TomAugspurger

@scollis just to verify, are you creating a Client object?

And are the panels blank, or do they have the Dask logo?

Eduardo Gonzalez
@eddienko
@scollis same here -- panels are blank, the Dask status pages contain only the top navbar
Scott
@scollis
Thanks @eddienko
@TomAugspurger Blank with a tab with the dask logo and the name, eg “Dask Task Stream”
Opening new panels does not help
This happens on your pangeo-notebooks example and my own “roll your own"
And @TomAugspurger I am running the examples in the pangeo-example-notebook repo
Tested now on Chrome and Safari
Jim Crist-Harif
@jcrist
@scollis , I've seen this behavior with tornado 6 and jupyter-server-proxy (the proxy used to proxy the dashboard through the user's notebook session). In the user's notebook environment, you need tornado 5 installed.
Scott
@scollis
Thanks.. will play with the environment.yml file
Joe Hamman
@jhamman
@rabernat, @rsignell-usgs, @tjcrone - can I merge this: pangeo-data/pangeo-cloud-federation#362
Scott Henderson
@scottyhq
@jhamman - just left a comment
Scott Henderson
@scottyhq
@fmaussion - we aren't updating images on any fixed timeline. But separating ‘staging’ and ‘production’ hubs has been useful to test things out and minimize disruption (see https://github.com/pangeo-data/pangeo-cloud-federation). As you already noted, the user home directory is unaffected. We’ve hit a lot of snags over the last months w/ version compatibilty when trying to have the latest packages (e.g. jupyterlab1, dask + dask-labextension). see https://github.com/pangeo-data/pangeo-stacks.
Joe Hamman
@jhamman
@mrocklin - if you’re around, let’s get you sorted on Cheyenne.
Scott
@scollis
Hey @scottyhq Thanks for the idea of using the base docker container.. I will give that a go…. Still lots to learn for me!
Scott
@scollis
All this stuff is very dfferent to your quickstarts.. if you want, once the dust settles from my learning I can put some PRs together with some new quickstarts including updating the cookie cutter
Scott Henderson
@scottyhq
no problem @scollis! thanks for trying things out. PRs are more than welcome. especially w/ documentation. it’s a bit of a moving target right now as we’re constantly experimenting with things so it's hard to keep current.
Scott
@scollis
so the base docker.. does that have the dask and jupyterlab set ups? so no need to include that anymore in the binder dir environment.yml
ie the environment.yml should now only have domain specific packages?
Scott Henderson
@scottyhq
that’s the idea yes. because we’re using repo2docker you end up layering packages that are defined in the following places in order 1) https://github.com/jupyter/repo2docker/blob/master/repo2docker/buildpacks/conda/environment.yml 2) https://github.com/pangeo-data/pangeo-stacks/blob/master/base-notebook/binder/environment.yml and 3) your own environment.yml.