Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 13 11:36
    stale[bot] closed #727
  • Dec 13 11:36
    stale[bot] commented #727
  • Dec 13 04:38
    rabernat commented #579
  • Dec 13 04:32

    rabernat on gh-pages

    Update docs after building Trav… (compare)

  • Dec 13 04:31
    rabernat synchronize #697
  • Dec 13 04:29

    rabernat on master

    Add JASMIN to deployments (#721… (compare)

  • Dec 13 04:29
    rabernat closed #721
  • Dec 12 14:45
    rabernat opened #749
  • Dec 12 09:42
    kaedonkers commented #721
  • Dec 11 22:56
    stale[bot] labeled #72
  • Dec 11 22:56
    stale[bot] commented #72
  • Dec 11 09:52
    kaedonkers synchronize #721
  • Dec 11 08:48
    kaedonkers commented #721
  • Dec 11 08:48
    kaedonkers commented #721
  • Dec 11 08:47
    kaedonkers synchronize #721
  • Dec 11 08:43
    kaedonkers synchronize #721
  • Dec 11 08:42
    kaedonkers synchronize #721
  • Dec 10 16:11
    rabernat commented #747
  • Dec 09 23:59
    Thomas-Moore-Creative commented #747
  • Dec 09 15:59

    rabernat on gh-pages

    Update docs after building Trav… (compare)

Frédéric BRIOL
@fbriol
The L3 altimetry data have been updated on the “pangeo-cnes” bucket. In this update, I modified the encoding settings to access the dataset more quickly. I used Zarr filters and other encoding options to significantly improve compression, and performance reading. More details here
Tim Crone
@tjcrone
I would like to open up ooi.pangeo.io to anyone with GitHub credentials during AGU. Does anyone know how to change and reload a Jupyterhub configuration without having to go through the CircleCI/Hubploy process? Any help would be greatly appreciated!
Joe Hamman
@jhamman
@tjcrone , we can go through hubploy, just make a PR directly to prod.
Tim Crone
@tjcrone
Okay roger that. Thanks @jhamman!
Joe Hamman
@jhamman
@yuvipanda, @consideratio, @scottyhq etc. I’m going to hang out here (https://whereby.com/pangeo) for a bit to talk about cloud ops.
Matthew Rocklin
@mrocklin
I was just about to ask for the location :)
Deepak Cherian
@dcherian
image.png
I'm getting a bunch of warning messages on ocean.pangeo.io. How do I turn these off?
ooh now the jlab server died. I guess there's some instability?
Matthew Rocklin
@mrocklin
Upgrade distributed I think
Deepak Cherian
@dcherian
yup distributed is at 2.7.0. I'll upgrade and try again
Ryan Abernathey
@rabernat
which image are you using?
Deepak Cherian
@dcherian
hmm.. CMIP I think? The second one on the list. Should I just always use the last one?
Ryan Abernathey
@rabernat
they use the same image
Yuvi Panda
@yuvipanda
connect to your JupyterHub from vscode (and you can use dask this way) https://blog.jupyter.org/connect-to-a-jupyterhub-from-visual-studio-code-ed7ed3a31bcb :)
Deepak Cherian
@dcherian
@yuvipanda nice!
Kevin Paul
@kmpaul
@yuvipanda :thumbsup:
Does anyone out there have any experience or knowledge of Arkouda? It appears to be Cray's Chapel-based competitor to Dask Arrays. ...And since Dask is open source, I struggle to understand why it was implemented in the first place.
Jim Crist-Harif
@jcrist
ooo, that looks very useful. Thanks for the link Yuvi.
Yuvi Panda
@yuvipanda
niiiceeee
Kevin Paul
@kmpaul
:thumbsup:
CopernicusUnive19
@CopernicusUnive19

hello everyone I'm new to geo spatial data science (1 week of work now).
I have a problem: I use geojson data to perform query with sentinelsat to retrieve sentinel images.

with this chunk of code footprint = geojson_to_wkt(read_geojson(GEOJSON_FILE)) I get the footprint with that I do he API call.
After retrieving the results and processing them with rasterio, I want to clip the image with my footprin, but with out_image, out_transform = rasterio.mask.mask(img, footprint ,crop=True) I get an AttributeError. I suspect that the problem are CRS. Is there a way to project my geojson file with the correct CRS in order to clip the image? thanks

David Hoese
@djhoese

@CopernicusUnive19 If you don't get a good answer from someone here I would check out the rasterio documentation and maybe contact them (they have a discussion forum/site thing) if you think it is a rasterio issue. I don't do much with geojson so not sure I can help.

Also note a lot of pangeo folks are at the AGU conference this week so they may be slow to respond.

Scott Henderson
@scottyhq
I recommend geopandas for vector geometries and reprojection http://geopandas.org/reference/geopandas.read_file.html http://geopandas.org/projections.html
We also included an intro to geo pandas notebook in our agu tutorial this year - https://github.com/pangeo-data/pangeo-tutorial/blob/agu2019/notebooks/geopandas.ipynb
Hope that helps!
CopernicusUnive19
@CopernicusUnive19
Thanks I solved by creating a geopandas dataframe and using the _to_crs() method
David Hoese
@djhoese

Does anyone know of an xarray-based project that has attempted to put DataArrays of different "resolutions" in to the same Dataset? Or if there is any work in xarray to handle cases like this better?

For example, in Satpy we might have an image of the Earth where each pixel represents 1km by 1km. We might have another image from the same instrument recorded at the same time but it is twice the size and each pixel represents 500m by 500m. I feel like @rabernat has done a lot with stuff like this if I remember correctly.

Ryan Abernathey
@rabernat
@djhoese - this is being actively discussed in zarr: zarr-developers/zarr-specs#23
I really encourage you to weigh in there with your use case
David Hoese
@djhoese
Thanks @rabernat. That's an interesting issue and good to know it is being worked on, but my issue is a little different. In my case the two resolutions of data are different variables that were recorded by the instrument at different resolutions, not the same variable at different resolutions.
Joe Hamman
@jhamman
@djhoese, I don’t think there is an established beat practice but I’ve been thinking about this. We need some sort of model/convention that supports this. The simple way is to just create multiple x/y dimensions for each grid resolution.
David Hoese
@djhoese
@jhamman That's what I figured. Thanks.
Joe Hamman
@jhamman
Hopefully tools built on top of xarray could provide additional functionality that harmonizes certain operations given a functional convention.
David Hoese
@djhoese
This came up for Satpy (it's always in my head) where we have this very high level Scene object that is stuck between being an "actor" object and a container. Right now it is essentially a dictionary of DataArrays. It would be much easier for non-satpy users to get access to some of our functionality if we could do things from an accessor or just accept an xr.Dataset as-is. The long term goal would be replacing the "Scene" object with a regular xr.Dataset.
Ryan Abernathey
@rabernat
ok I think I understand better now
So you want a data model that understands cell geometry and can resample / broadcast appropriately
I think this is a very common need
James A. Bednar
@jbednar
Yep.
David Hoese
@djhoese
@rabernat I know that your use cases (from the packages I've seen you link to and your presentations) are much more complicated than mine where typically I have one center point for a pixel and that is good enough in most cases
Deepak Cherian
@dcherian

Hopefully tools built on top of xarray could provide additional functionality that harmonizes certain operations given a functional convention.

yeah like .sum(dim="X") should sum each variable over it's respective "X" dimension. Sounds like a job for cf-xarray

(mythical cf-xarray)
Ryan Abernathey
@rabernat
:pray:
Joe Hamman
@jhamman
@djhoese - I think this conversation would make a great discourse topic! I’d love to see it continue but fear it will be lost in the ether if it continues here.
Tina Odaka
@tinaok
hi @jhamman , (responding to your message on 14th Nov, i’m sory for my very late response...) the presentation on interactive HPC workshop at SC19 ( https://sites.google.com/view/interative-hpc/home) , is availble from here https://drive.google.com/file/d/1zLMH7CGWRXgpi6eanwW0X4ZvdNfN9mwc/view
David Hoese
@djhoese
@dcherian I was kind of hoping the geoxarray project I started would do that, but I just can't bring myself to make any hard decisions about how things should be implemented
@jhamman If I have time I'll make something today
David Hoese
@djhoese
@rabernat You were right about TileDB support. I am really impressed by their responsiveness to my questions and bug reports.