Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 12 14:45
    rabernat opened #749
  • Dec 12 09:42
    kaedonkers commented #721
  • Dec 11 22:56
    stale[bot] labeled #72
  • Dec 11 22:56
    stale[bot] commented #72
  • Dec 11 09:52
    kaedonkers synchronize #721
  • Dec 11 08:48
    kaedonkers commented #721
  • Dec 11 08:48
    kaedonkers commented #721
  • Dec 11 08:47
    kaedonkers synchronize #721
  • Dec 11 08:43
    kaedonkers synchronize #721
  • Dec 11 08:42
    kaedonkers synchronize #721
  • Dec 10 16:11
    rabernat commented #747
  • Dec 09 23:59
    Thomas-Moore-Creative commented #747
  • Dec 09 15:59

    rabernat on gh-pages

    Update docs after building Trav… (compare)

  • Dec 09 15:57

    jhamman on master

    Update meeting-notes.rst (compare)

  • Dec 09 15:10
    mrocklin commented #747
  • Dec 09 14:18

    rabernat on gh-pages

    Update docs after building Trav… (compare)

  • Dec 09 14:16

    rabernat on master

    More informative landing page (… (compare)

  • Dec 09 14:16
    rabernat closed #748
  • Dec 09 10:01
    mkjpryor-stfc commented #721
  • Dec 09 06:30
    Thomas-Moore-Creative commented #747
Scott
@scollis
I guess I could push the data back in the futures and then loop over them and dispose of the data as I save..
Let me experiment with that.. fun :)
Pier
@PhenoloBoy
@scollis Hi, in which format are you going to save your data once back from the workers?
Scott
@scollis
netcdf.
I know.. I really need to learn XARR
Pier
@PhenoloBoy
can you process your data per lines?
Scott
@scollis
Its very nicely parallel.. a task per time step. So chunking by time is easy..
Pier
@PhenoloBoy
I've got to solve a similar problem and, in my case, the only solution has been an append results in a netCDF4 file
netcdf not xarray
Scott
@scollis
Cool. That’s the kind of thing I am looking at doing
Pier
@PhenoloBoy
you have to create an empty netCDF http://unidata.github.io/netcdf4-python/netCDF4/index.html and almost that's it
even if isn't a super clean method in my case has been the only solution
but be aware that netcdf has some problem over GC
so before you start with this approach have a look at some discussion about netCDF and the Cloud base infrastructures
Scott
@scollis
Will do.. I am just messing around right now but will be hacking at this more seriously soon
Anyone got some cool code that shows dealing with futures as they complete on dask?
Pier
@PhenoloBoy
let me see if I've anything, most of the time I'm using that approach
Scott
@scollis
Thanks!
This is my loop once done script I want to have run while the compute is ongoing
tpls = [] for this_future in future: gathered = client.gather(this_future) pyart.io.write_grid(gathered[-1], gathered[-2]) tpls.append(tpls[0:-3]) del gathered
hmm.. not so good for showing code
:D
Pier
@PhenoloBoy
don't worry it's enough
have a look to this, even if is far fro be perfect and is more written by a monkey it could help you. You have to readapt as the writing part isn't there. Unfortunately, I couldn't retest if it's working as I'm a little bit busy
Scott
@scollis
Awesome, thanks!
Pier
@PhenoloBoy
the approach is unconventional and I don't suggest to anybody to follow it. Time to time in some cases is the only solution that I figured out
Scott
@scollis
@jhamman is there any example that stores data from Kuberneties workers to a cloud store like google cloud as a way of returning data?
Pier
@PhenoloBoy
Zarr or parquette is your solution
Scott
@scollis
Thats really nice @PhenoloBoy … acts as way to start thinking about stuff.. Yeah.. gotta learn to use Zarr
Pier
@PhenoloBoy
I've made the same question to @jhamman few days ago
the solution is to use Zarr or Parquet
seems that in the upcoming 6 months there will be a beta for netCDF that will use Zarr but right now is more gossip than anything else ( at least for my understanding)
Rob Fatland
@robfatland
@rabernat fantastic i was hoping you'd be available.
Charles Blackmon-Luca
@charlesbluca
Quickly tossed together some notebooks to generate a catalog of all the data on gs://pangeo-data; my idea is that one day a script could be automated to do this on a regular basis
Rob Fatland
@robfatland
If nobody minds I'm going to hijack appear.in/pangeo from 5pm to 5:30pm PDT today for a conversation on Megaptera, our citizen science whale call identification ML project. LMK if any conflicting and I'll bang on over to zoom.
Anderson Banihirwe
@andersy005

RE: For some reason, @tjcrone and I are have trouble making intake work with s3. I think Tim will open an intake issue.

@rabernat & @tjcrone, did you figure this out? Could you expand on what the exact issue was? I am trying to create static intake catalogs pointing to CESM LENS data in S3 and I seem to be having some issues when accessing the data.

Joe Hamman
@jhamman
@robfatland :thumbsup:
Tom Augspurger
@TomAugspurger
Ping me if you’re having issues with s3fs. There’s been some churn lately.
Filipe
@ocefpaf
Did anyone loose a laptop charger during the meeting last week?
Ryan Abernathey
@rabernat
@tjcrone - could you open an intake issue about the s3fs / intake problem we ran into? I don't have the code to reproduce on my machine.
Ryan Abernathey
@rabernat
Satpy / pyresample experts. What is the best way to serialize an area definition and store it in an xarray dataset?
Satpy does some of this stuff internally, but I want to roll my own.
@djhoese
Philip Austin
@phaustin
@ocefpaf pretty sure that's my mac charger -- feel free to gift it to some visitor/grad student who finds themself without one.
Rob Fatland
@robfatland
@ocefpaf I also lost a charger: Black Surface charger, has a blade-like charger that clips on via magnet; also has a spare USB output port
Ryan Abernathey
@rabernat
ocean.pangeo.io seems to be down
I just added a node to the core pool
Yuvi Panda
@yuvipanda
I am going to submit a PANGEO talk to the local kubernetes meetup here
Primarily as a recruiting tool :D