But if you want the data locally, you need to compute first.
I guess I could push the data back in the futures and then loop over them and dispose of the data as I save..
Let me experiment with that.. fun :)
I know.. I really need to learn XARR
netcdf not xarray
you have to create an empty netCDF http://unidata.github.io/netcdf4-python/netCDF4/index.html and almost that's it
even if isn't a super clean method in my case has been the only solution
but be aware that netcdf has some problem over GC
so before you start with this approach have a look at some discussion about netCDF and the Cloud base infrastructures
Anyone got some cool code that shows dealing with futures as they complete on dask?
This is my loop once done script I want to have run while the compute is ongoing
tpls = []
for this_future in future:
gathered = client.gather(this_future)
pyart.io.write_grid(gathered[-1], gathered[-2])
tpls.append(tpls[0:-3])
del gathered
hmm.. not so good for showing code
:D
have a look to this, even if is far fro be perfect and is more written by a monkey it could help you. You have to readapt as the writing part isn't there. Unfortunately, I couldn't retest if it's working as I'm a little bit busy
the solution is to use Zarr or Parquet
seems that in the upcoming 6 months there will be a beta for netCDF that will use Zarr but right now is more gossip than anything else ( at least for my understanding)
Quickly tossed together some notebooks to generate a catalog of all the data on gs://pangeo-data; my idea is that one day a script could be automated to do this on a regular basis
If nobody minds I'm going to hijack appear.in/pangeo from 5pm to 5:30pm PDT today for a conversation on Megaptera, our citizen science whale call identification ML project. LMK if any conflicting and I'll bang on over to zoom.
RE: For some reason, @tjcrone and I are have trouble making intake work with s3. I think Tim will open an intake issue.
@rabernat & @tjcrone, did you figure this out? Could you expand on what the exact issue was? I am trying to create static intake catalogs pointing to CESM LENS data in S3 and I seem to be having some issues when accessing the data.
Satpy does some of this stuff internally, but I want to roll my own.