Heya. I've got a 3TB dataset that I'm writing out via xarray+zarr+gcsfs. The chunking is O(100MB), but I've also tried smaller. I am using ~1000 workers. I am running into some weird errors that I'm having trouble tracking down (dask/gcsfs#288). They manifest as a path that becomes None with gcsfs.
I have to imagine this has been done before. I'm curious if it's a version problem, and someone might have a working setup like this and can share their requirements/environment.