by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Aug 08 07:25
    stale[bot] closed #764
  • Aug 08 07:25
    stale[bot] commented #764
  • Aug 01 08:33
    stale[bot] closed #778
  • Aug 01 08:32
    stale[bot] commented #778
  • Jul 29 19:08

    TomAugspurger on user-install-caveats

    (compare)

  • Jul 29 19:08

    TomAugspurger on user-install

    (compare)

  • Jul 29 19:07

    TomAugspurger on user-install-caveats

    more caveats (compare)

  • Jul 29 19:07

    TomAugspurger on user-install-caveats

    Merge remote-tracking branch 'o… more caveats (compare)

  • Jul 29 19:01

    rabernat on gh-pages

    Update docs after building Trav… (compare)

  • Jul 29 18:59

    rabernat on master

    DOC: Demonstrate user installs … (compare)

  • Jul 29 18:59
    rabernat closed #782
  • Jul 29 18:33
    TomAugspurger opened #782
  • Jul 29 18:21

    TomAugspurger on user-install

    DOC: Demonstrate user installs (compare)

  • Jul 27 18:13
    stale[bot] labeled #764
  • Jul 27 18:13
    stale[bot] commented #764
  • Jul 25 19:39
    stale[bot] closed #777
  • Jul 25 19:39
    stale[bot] commented #777
  • Jul 25 06:40
    stale[bot] closed #752
  • Jul 25 06:40
    stale[bot] closed #776
  • Jul 25 06:40
    stale[bot] closed #755
Joe Hamman
@jhamman
hmmm, for your info, I hadn’t done anything in my session yet. It just created then died almost immediately.
Scott Henderson
@scottyhq
not exactly sure what fills it up, but looks like all the nodes are still launching w/ default 20Gb EBS disk (new clusters we updated to 100Gb)
Scott Henderson
@scottyhq
in fact we documented this earlier! pangeo-data/pangeo-cloud-federation#274
Scott Henderson
@scottyhq
i’ll have a bit of time this afternoon to make sure its updated
David Hoese
@djhoese

@jhamman At scipy you talked about how pangeo's binder can scale down to 0. Any idea how long it keeps things alive before shutting everything down? What about time to start up? What about timeout on individual inactive JLab sessions?

I was thinking if I load a repository X minutes before a tutorial then everyone should have speedy access to their JLab session. However, if I do it too early then we'll have to wait anyway.

Joe Hamman
@jhamman
I think the jupyter session timeout is something like 60 minutes. K8s nodes scale down after that and I don’t recall the rate. You would certainly have 5-10 minutes though.
David Hoese
@djhoese
thanks
Kolmar Kafran
@kafran
'{:,.2f}'.format(1234.1234) results 1,234.12. How can I get 1.234,12 since '{:.,2f}'.format(1234.1234)is not valid?
David Hoese
@djhoese

@kafran You may want to look at the locale module and the n string format specifier. Have you tried stackoverflow? Best answer I found was:

In [32]: '{:.2n}'.format(1234.1234)                                                                                                        
Out[32]: '1,2e+03'

Which I think is a bug in python since it shouldn't be using exponent formatting

Nevermind, that might be intended but I still can't find a way to do what you want
if you post a question on stackoverflow please link it here. I'm curious
matrixbot
@matrixbot
ashwinvis Kolmar Kafran (Gitter)Kolmar Kafran (Gitter) Using comma as decimal might depend on locale settings. If your are on linux try localectl status
Kolmar Kafran
@kafran
@matrixbot I already tried it
Scott Henderson
@scottyhq
@philippjfr - staging.nasa.pangeo.io is back up if you want to keep experimenting
Sarah Bird
@birdsarah
quick poll in case anyone has experience - how steep is the learning curve (by your own definition) for kafka?
Matthew Rocklin
@mrocklin
Steep hill that is short enough that you can jog up it. Then it plateaus for a bit. Then there is a climb later on if you feel like going forward. There isn't really much to it as a system if you're just using it as storage. Also, some random notes on the Python APIs: http://matthewrocklin.com/blog/work/2017/10/10/kafka-python
Sarah Bird
@birdsarah
thanks @mrocklin
David Hoese
@djhoese
Is anyone up to date on what Unidata's work for NetCDF and zarr compatibility actually entails? Mainly how I, the user, would interact or use this functionality? I'm guessing all GOES-16 ABI NetCDF files on GCS and S3 would have to be rewritten in zarr format and that format could be read by NetCDF? Or...something? Anyone know?
James A. Bednar
@jbednar
@rsignell-usgs would probably know...
Dr. Andreas Hopfgartner
@anderl80
Hi all, I have a Pangeo Cluster in my gcloud. I'm not so familiar with gcloud (more Azure), is there a possibility to shut down the cluster for having no costs?
Matthew Rocklin
@mrocklin
Dask Jobqueue + SpecCluster rewrite issue here, if anyone wants to play around with HPC systems: dask/dask-jobqueue#306
Ryan May
@dopplershift
@djhoese The work going in netcdf-c will make zarr work as 'just' another on-disk format for the library to use, like it can use HDF5 or netCDF3. Really, though, this is a great question to send to the netCDF user list: https://www.unidata.ucar.edu/software/netcdf/mailing-lists.html (or support-netcdf@unidata.ucar.edu)
@djhoese As far as the current data in S3 (and maybe GCS), there is support in the library now to make byte-range requests over HTTP, so I think you could do a direct read of hdf5-backed data in S3 using the library.
David Hoese
@djhoese
@dopplershift any idea how it encodes that in the HTTP request? Doesn't that have to be understood by the server? I guess that is important enough that it was probably put in a while ago
Ryan May
@dopplershift
Byte ranges are a standard http request header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
Tom Augspurger
@TomAugspurger

Is anyone here using ocean.pangeo.io heavily right now? pangeo-data/pangeo-cloud-federation#360. Trying to figure out why it’s giving a 504.

It looks like there’s a bunch of dask worker pods running. Trying to figure out the hub isn’t handling new users though.

Joe Hamman
@jhamman
hey @TomAugspurger. I’m here now
Eduardo Gonzalez
@eddienko

I have deployed pango in my local cluster (thanks for making it easy!). I do have question about the dask integration. I have a few volumes containing data that are mounted in the jupyter notebook server and defined in the cluster specific configuration jupyter_config.yaml (using extraVolumes and extraVolumeMounts). So far so good.

The problem is that when launching dask using KubeCluster from the notebook, these volumes are not available from the dask workers. Is there an 'automatic' way of doing this or do I need to define the volumes as well in the dask.yaml file? Or am I gettting this wrong (possibly!)?

Joe Hamman
@jhamman
Glad to hear it’s going well so far. Yes, you need to mount the volumes in your workers as well.
Eduardo Gonzalez
@eddienko
Ok, thanks. EXTRA_PIP_PACKAGES is not propagated neither. I wrote this in case it is useful: https://gist.github.com/eddienko/7b5b40ff18309ecd7f28d294c3f1cd72
Joe Hamman
@jhamman
I think the EXTRA_PIP_PACKAGES option is no more. Where did you find that?
Eduardo Gonzalez
@eddienko
This is linked from the official Pangeo.io documentation: https://github.com/pangeo-data/pangeo/blob/master/gce/setup-guide/jupyter_config.yaml
Ryan Abernathey
@rabernat
How can we get diagnostics on our google cloud firestore?
like disk usage, etc.
Joe Hamman
@jhamman
No idea but this seems useful.
Ryan Abernathey
@rabernat
I'm worried that ocean hack week users are going to consume a lot of home storage space on ocean.pangeo.io
it would be good to have some monitoring
Scott
@scollis
Hey Folks.. Thanks to some help from @jhamman I am up and running. I was running into an issue that I did not get the nice views on the panels when I was running Dask.. (profiler et al).. So I ran the example Pangeo notebooks using the binder link. http://binder.pangeo.io/v2/gh/pangeo-data/pangeo-example-notebooks/master and I get the same issue that I never get nice output..even when I click the dask icon in the right bar I can not see what workers are doing.. Could there be a network blocking issue (some stuff is blocked here at Argonne)?
is blank
Wonder if it is a Bokeh issue...
Fabien Maussion
@fmaussion
Hi Pangeo folks: how often do you actualize the base image on hub.pangeo.io (or do you even do that?)
What are the consequences for the users? I expect these to be minimal, i.e. a user's HOME is completely decoupled from the base image
but still, what real bad could happen?
Scott
@scollis
So I tried on my home computer but I still get blank panels.. Just curious to see if this on only me (browser: Chrome, OS: MacOS)
Tom Augspurger
@TomAugspurger

@scollis just to verify, are you creating a Client object?

And are the panels blank, or do they have the Dask logo?

Eduardo Gonzalez
@eddienko
@scollis same here -- panels are blank, the Dask status pages contain only the top navbar
Scott
@scollis
Thanks @eddienko