Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Nov 27 12:01
    willirath opened #805
  • Nov 19 06:30
    RichardScottOZ commented #744
  • Nov 16 15:07
    ocefpaf commented #792
  • Nov 16 15:03
    wolfv commented #792
  • Nov 16 14:09
    jbusecke commented #792
  • Nov 16 03:57
    stale[bot] labeled #784
  • Nov 16 03:56
    stale[bot] commented #784
  • Nov 14 22:17
    ocefpaf commented #792
  • Nov 14 02:49
    scottyhq commented #792
  • Nov 14 02:49
    scottyhq commented #792
  • Nov 13 23:37
    jbusecke commented #792
  • Nov 13 23:30
    dcherian commented #792
  • Nov 13 23:06
    jbusecke commented #792
  • Nov 13 19:39
    AJueling commented #351
  • Nov 12 02:08
    jbusecke commented #804
  • Nov 11 14:50

    rabernat on gh-pages

    Update docs after building Trav… (compare)

  • Nov 11 14:16

    rabernat on master

    Update index.rst (#803) * Upda… (compare)

  • Nov 11 14:16
    rabernat closed #803
  • Nov 11 13:33
    lhoupert synchronize #803
  • Nov 11 13:16

    rabernat on gh-pages

    Update docs after building Trav… (compare)

Chiara Lepore
@chiaral
mm i can't find it on the instruction page anymore - but anyways - to check on the size of the cluster with the
from dask_gateway import Gateway
g = Gateway()
g.list_clusters()
and then connect to your cluster and check it out - mine was up to 30 workers but I had requested only 18... so sorry!
Chelle Gentemann
@cgentemann
no worries! thanks for checking!
Rich Signell
@rsignell-usgs
weekly telecon today at 12 PM EST, right? (just want to make sure it's not 11AM EST)
James A. Bednar
@jbednar
12 EST is what I have.
Ryan Abernathey
@rabernat
yes
Rich Signell
@rsignell-usgs
Great. Also, I knew Tom worked on Pangeo and Dask, but wow... https://tomaugspurger.github.io/whats-next.html
Tom Augspurger
@TomAugspurger
Lots of small roles on all those.
James A. Bednar
@jbednar
You're too modest. :-)
Ryan Abernathey
@rabernat
Tom is our 10x engineer! 🙌
Ryan Abernathey
@rabernat
Weekly checkin meeting happening in 5 minutes!
epifanio
@epifanio
Hi! not sure where to ask .. but I know you have experience with running python scientific stack on the cloud, especially K8S .. :)
I am dealing with a pod running on a read-only filesistem where my app based on xarray+bokeh &co need to run
I am facing this error: https://gist.github.com/dac53dde203cfbc77f143f1c54f45496
HAve evere had a similar issue?
It seems the .pyc files can not be generated (along with bokeh cookies, but seems the problem is with the netcdf4 library)
Any advice :) ?
James A. Bednar
@jbednar
Not sure; do the file permissions on the .nc file allow writing?
epifanio
@epifanio
the nc file is coming from an url to opendap - the app works fine in a docker-swarm setup: https://ncplot.epinux.com/test/TS-Plot?url=https://hyrax.epinux.com/opendap/ctdiaoos_gi2007_2009.nc it takes an url ah parameter to then guess how to render it
James A. Bednar
@jbednar
No idea, then.
epifanio
@epifanio
my only guess is that: OSError: [Errno -37] NetCDF: Write to read only: b'https://hyrax.epinux.com/opendap/ctdiaoos_gi2007_2009.nc' <-- means Netcdf4 is trying to cache something
Wesley Banfield
@WesleyTheGeolien

@epifanio I wonder if the python interpreter is writing cache .pyc files?

You can disable it by running python with -B or apparently this works

import sys

sys.dont_write_bytecode = True

https://stackoverflow.com/questions/154443/how-to-avoid-pyc-files

Rich Signell
@rsignell-usgs
If I'm having an issue with a rechunker workflow, better to ask on Pangeo discourse, or raise an Github issue on rechunker?
Ryan Abernathey
@rabernat
Raise issue on rechunker please
Wesley Banfield
@WesleyTheGeolien

Hey,

Does anyone have any ideas on how to export a function as a literal string (sort of a reverse eval)? I am trying to create a script which reproduces steps and exports to a separate python file.

I know I could just copy paste the contents but not very good on the DRY front ...

Martin Durant
@martindurant
inspect.getsource ?
Wesley Banfield
@WesleyTheGeolien
exactly what I was looking for! I was not aware of it's existance! Cheers @martindurant
epifanio
@epifanio

@WesleyTheGeolien thanks for the hints. I tried to load a simpler bokeh-baked app .. and it was actually working (except for the fontawesome icon which required on the fly build) -- so I made sure it is not a matter of writing .pyc files (I also used the environament var:

        env:
        - name: PYTHONDONTWRITEBYTECODE
          value: "True"

)
So I think is a xarray thing, which has by default a cache option - I turned it to off with no success.
I am now trying to use something like:
https://github.com/pydata/xarray/blob/34aa056320654868e1f4591e1c8f466bae85efb7/xarray/tutorial.py#L32
I see the cache_dir is not an option in the xarray version I have

epifanio
@epifanio
I asked sys admin to temporary remove the readonly restriction .. and the app was working .. so I guess I need to find the path used by default by xarray (or netcdf4) .. to write down those tmporary files
and expose it as volume on the host
epifanio
@epifanio
ok, seems I got it to work :) was a pain to figure out the K8S settings to do that ... but exposing /tmp as a volumemount of type emptydir .. did the trick
thank god was a default /tmp ...
Rich Signell
@rsignell-usgs
@epifanio, Did you figure out what package was writing there? Seems like caching is off by default for netcdf4? https://github.com/Unidata/netcdf4-python/issues/551#issuecomment-486171545
epifanio
@epifanio
i was using xarray, but the error log was referring to netcdf4
which I guess is used under the hood by xarray as defasult engine to read the opendap netcdf url
epifanio
@epifanio
I tried to set cache=False in xr.open_dataset() guessing it was transfering the option to necdf4, but that didn't do the trick
Rich Signell
@rsignell-usgs
@epifanio maybe raise an issue on xarray?
epifanio
@epifanio
yes, sounds a good Idea to keep track it - thanks
James A. Bednar
@jbednar
Tom's old job at Anaconda (or at least the Dask part of it) is open if anyone wants to try to fill his shoes: https://boards.greenhouse.io/anaconda/jobs/2451164
Ryan Abernathey
@rabernat
James A. Bednar
@jbednar
Thanks; done!
Beto
@betolink
Hey folks, I work at a NASA DAAC and wanted to know if there is guidance on how to use your Binderhub deployment, from the administrative point of view. Who can use it, how many concurrent users can handle etc. Is there any documentation on this?
Ryan Abernathey
@rabernat
Hi Beto. For now, anyone can use it, and there are no hard limits.
Everything technical about the binder deployment is described here: https://github.com/pangeo-data/pangeo-binder
If you have a more specific question, an issue on that repo would be a good place to ask.
Joe Hamman
@jhamman
@betolink - we have also been asking people to post on this issue (pangeo-data/pangeo#440) if they will be using any of the public pangeo deployments for larger educational events.
Beto
@betolink
Thanks @rabernat and @jhamman! there is a ~3 hour tutorial NASA scientists want to present at AGU. They think they'll have around 80 people using intake, xarray etc (not Dask) on small data (< 1gb of data per user). I suggested that we should consider using Pangeo's Binderhub to speed things up so they can focus on the content and not the infrastructure.
3 replies
Yuvi Panda
@yuvipanda
@jhamman For the future, I'm wondering if 2i2c can use some of the PANGEO credits to run short-term hubs like these
@scottyhq @rabernat might also have ideas
Anne Fouilloux
@annefou
Maybe not useful/relevant but just to let you know that in Europe, Galaxy Europe provides Training infrastructure as a service (https://galaxyproject.eu/tiaas.html) and we have a Pangeo JupyterLab as an interactive tool (https://live.usegalaxy.eu/). I amthinking to add a Pangeo Binder as an interactive tool. But maybe it is silly.
6 replies
Takaya Uchida
@roxyboy
Hi all, I've been trying to set up a Pangeo-binderhub but everytime I call gateway.new_cluster(), I get a GatewayClusterError: Cluster failed to start, see logs for more information. Has anyone else faced this issue?
Erik Sundell
@consideRatio
@roxyboy if the JupyterHub helm chart version 0.10+ is involved, then disable network policies for the hub. hub.networkPolicy.enabled=false.
Takaya Uchida
@roxyboy
Thank you @consideRatio for the information. Could you advise in which file I should put this command? As of now, I've tried putting it in binder/config.yml
Wesley Banfield
@WesleyTheGeolien
@roxyboy is there a reason you are setting up a BinderHub of JupyterHub (with pangeo stuff installed) I am curious because just about to look into setting one up myself and I though the Binder side would make things less persistent
Takaya Uchida
@roxyboy
@WesleyTheGeolien I'm using data stored on the Pangeo allocation of Google Cloud so using the Pangeo Binderhub allows for reproducibility of the analysis.
2 replies