pangeo-data/Lobby

general Pangeo discussion

Pier

@PhenoloBoy

even if isn't a super clean method in my case has been the only solution

but be aware that netcdf has some problem over GC

so before you start with this approach have a look at some discussion about netCDF and the Cloud base infrastructures

Scott

@scollis

Will do.. I am just messing around right now but will be hacking at this more seriously soon

Anyone got some cool code that shows dealing with futures as they complete on dask?

Pier

@PhenoloBoy

let me see if I've anything, most of the time I'm using that approach

Scott

@scollis

Thanks!

This is my loop once done script I want to have run while the compute is ongoing

tpls = []
for this_future in future:
    gathered = client.gather(this_future)
    pyart.io.write_grid(gathered[-1], gathered[-2])
    tpls.append(tpls[0:-3])
    del gathered

hmm.. not so good for showing code

Pier

@PhenoloBoy

don't worry it's enough

have a look to this, even if is far fro be perfect and is more written by a monkey it could help you. You have to readapt as the writing part isn't there. Unfortunately, I couldn't retest if it's working as I'm a little bit busy

https://www.dropbox.com/s/2regfc2u5bgk9fs/cyclic_test.ipynb?dl=0

Scott

@scollis

Awesome, thanks!

Pier

@PhenoloBoy

the approach is unconventional and I don't suggest to anybody to follow it. Time to time in some cases is the only solution that I figured out

Scott

@scollis

@jhamman is there any example that stores data from Kuberneties workers to a cloud store like google cloud as a way of returning data?

Pier

@PhenoloBoy

Zarr or parquette is your solution

Scott

@scollis

Thats really nice @PhenoloBoy … acts as way to start thinking about stuff.. Yeah.. gotta learn to use Zarr

Pier

@PhenoloBoy

I've made the same question to @jhamman few days ago

the solution is to use Zarr or Parquet

seems that in the upcoming 6 months there will be a beta for netCDF that will use Zarr but right now is more gossip than anything else ( at least for my understanding)

Rob Fatland

@robfatland

@rabernat fantastic i was hoping you'd be available.

Charles Blackmon-Luca

@charlesbluca

https://github.com/pangeo-bot/pangeo-datastore

Quickly tossed together some notebooks to generate a catalog of all the data on gs://pangeo-data; my idea is that one day a script could be automated to do this on a regular basis

Rob Fatland

@robfatland

If nobody minds I'm going to hijack appear.in/pangeo from 5pm to 5:30pm PDT today for a conversation on Megaptera, our citizen science whale call identification ML project. LMK if any conflicting and I'll bang on over to zoom.

Anderson Banihirwe

@andersy005

RE: For some reason, @tjcrone and I are have trouble making intake work with s3. I think Tim will open an intake issue.

@rabernat & @tjcrone, did you figure this out? Could you expand on what the exact issue was? I am trying to create static intake catalogs pointing to CESM LENS data in S3 and I seem to be having some issues when accessing the data.

Joe Hamman

@jhamman

@robfatland :thumbsup:

Tom Augspurger

@TomAugspurger

Ping me if you’re having issues with s3fs. There’s been some churn lately.

Filipe

@ocefpaf

Did anyone loose a laptop charger during the meeting last week?

Ryan Abernathey

@rabernat

@tjcrone - could you open an intake issue about the s3fs / intake problem we ran into? I don't have the code to reproduce on my machine.

Ryan Abernathey

@rabernat

Satpy / pyresample experts. What is the best way to serialize an area definition and store it in an xarray dataset?

Satpy does some of this stuff internally, but I want to roll my own.

@djhoese

Philip Austin

@phaustin

@ocefpaf pretty sure that's my mac charger -- feel free to gift it to some visitor/grad student who finds themself without one.

Rob Fatland

@robfatland

@ocefpaf I also lost a charger: Black Surface charger, has a blade-like charger that clips on via magnet; also has a spare USB output port

Ryan Abernathey

@rabernat

ocean.pangeo.io seems to be down

I just added a node to the core pool

Yuvi Panda

@yuvipanda

I am going to submit a PANGEO talk to the local kubernetes meetup here

Primarily as a recruiting tool :D

Probably do the sea level raise demo, and phrase it like 'fight climate change with kubernetes' or something like that

Yuvi Panda

@yuvipanda

and then hopefully I can go around giving this talk in many places

I know this isn't all that PANGEO does but it's a good sell

Philip Austin

@phaustin

@yuvipanda — if you wind up with an AMI/gce image for live demos like this we would definitely be interested in reusing it — I gave a Math Dept. Jupyter Day pangeo talk yesterday using TLJH on EC2 and @mrocklin ’s https://www.youtube.com/watch?v=nH_AQo8WdKw notebook as a dask motivator and it went smoothly.

Ryan Abernathey

@rabernat

@yuvipanda :+1:

Yuvi Panda

@yuvipanda

submitted it! I've called myself a 'part of the PANGEO project', which I hope isn't overstating it!

@phaustin will do! I think I'll mostly cannibalize other folks' talks, but tune them to a devops audience

with that my keyboard time for the day is up! ttyl!

Ryan Abernathey

@rabernat

you are definitely "part of the PANGEO project"! In fact, it's hard to imagine where we would be without you!

Ian Rose

@ian-r-rose

:arrow_double_up:

David Hoese

@djhoese

@rabernat Check out my work on geoxarray: geoxarray/geoxarray#13

The way I see it you have a couple choices. Mainly depends on what your goal is for the representation/serialization.

If we are talking about the best way to make a serializable version of a grid/area definition, your main issue is the projection. Well Known Text (WKT) is supposed to be the most fully defined way of describing a projection. PROJ strings apparently can't fully describe all projections. I've been leaning towards pyproj's CRS objects as my preferred container for projection information since it can convert to PROJ strings, PROJ dicts, WKT (different versions), or a CF grid mapping variable. There was also a discussion between @snowman2 and @dopplershift about seeing how possible it would be to use pyproj's CRS objects to replace cartopy's CRS objects (or at least making cartopy's based on pyproj's).

You then have your extent information. Do you force them all to be the same or allow for all the variations? lower-left + upper-right + number of rows/columns, upper-left + pixel size + number of rows/columns, some other combination of these types of parameters, etc.

Are you going to define a single object that defines this information (pyresample's AreaDefinition) or encode it entirely in the xarray Dataset? If in the Dataset, do you depend on the x and y coordinates to define the extents? Or do you have separate attributes/coordinates?

Where communities thrive

People

Repo info

Activity