Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Rich Signell
    @rsignell-usgs
    Another thing I thought of: although the NetCDF library is adding support for Zarr (e.g. reading Zarr with the NetCDF library), it won't likely be as efficient as reading NetCDF4 with the Zarr library (using our approach)! Because it will still read the metadata piecemeal. How ironic, right?
    Martin Durant
    @martindurant
    It is also true that the blocks of a netcdf would not be read concurrently in any library except via our method, I think. This is important in the limit of small chunks.
    Rich Signell
    @rsignell-usgs
    Oh, I didn't think of that. Yes, there is a Parallel NetCDF project, but that requires extra work on the part of the user and is not directly supported by Unidata. So we make it easy to read NetCDF in parallel, and with standard python libraries supported by the community!
    Martin Durant
    @martindurant
    “Concurrently” :). Parallelism with dask would already work with netCDF, thread-wise. But for small chunks, you would still pay the latency cost a lot.
    I think the pont of having zarr as an official backend for netcdf is that it encourages new data to be written into zarr. We’re happy with that - an actual cloud-friendly format/library trumps our tricks.
    Rich Signell
    @rsignell-usgs
    Ah, okay, you mean the async, right?
    Martin Durant
    @martindurant
    Right. I call this “concurrency” usually. Threads also run async (i.e., command control timing is not guaranteed) without the async python keyword. The technical difference is between cooperative multitasking and preemptive multitasking.
    Lucas Sterzinger
    @lsterzinger
    @rsignell-usgs @martindurant Thanks for your help and clarification! I've struggled explaining exactly /why/ NetCDF performs poorly in the cloud as I wasn't super familiar with the details of the library/format. I agree with martin in that the biggest benefit (as a user) that I've seen using reference filesystem is the multi-file virtual dataset -- a process that takes forever with traditional xarray/netcdf workflows.
    Lucas Sterzinger
    @lsterzinger

    For the “extra storage” of the references, you might want to note that there are various encoding tricks that work well, the simplest would be to zstd compress the whole json: maybe gets you a factor of 10 in size, but is super fast to unpack.

    This is being given to a group with only basic python knowledge and so I was really focused on just showing a concept, not optimization/storage tricks

    Rich Signell
    @rsignell-usgs
    Lucas, to me it's important you don't say that you need to read the whole netcdf4 file to get the metadata.
    And also note that the compressed chunks of data in NetCDF4 can be extracted just as cloud-friendly as Zarr format
    Lucas Sterzinger
    @lsterzinger

    Lucas, to me it's important you don't say that you need to read the whole netcdf4 file to get the metadata

    Yes I agree. This is a piece of information I got from a conversation with Kevin Paul at NCAR -- I think I either misunderstood what he was saying (the most likely reason) or he misunderstood exactly how cloud access happens... maybe thinking of a time where you could only request entire files from object storage.

    I will make updates to the presentation before the workshop, thank you for your help!

    Rich Signell
    @rsignell-usgs
    :thumbsup:
    Good luck with the workshop! May your clusters spawn!
    I was joking with Zac Flamig we might call our framework "Fragile Architecture for Reproducible and Transparent Science"
    Martin Durant
    @martindurant
    Farts?
    Rich Signell
    @rsignell-usgs
    :thumbsup:
    I'm about to turn 60 and still making fart jokes. That's good, right?
    :)
    Martin Durant
    @martindurant
    It never gets old
    By the way @rsignell-usgs , you mentioned possible funding when I was talking about extra viz tools for our kind of big datasets, e.g., to know which parts of the satellite imagery contain data. JimB has a person that might have hours available for that kind of thing.
    Chelle Gentemann
    @cgentemann
    I wake up to FARTS. love you all. :)
    Rich Signell
    @rsignell-usgs
    @martindurant , okay, I'll see what I can do
    @cgentemann , phew! (or should I say "piu"!)
    Martin Durant
    @martindurant
    I was initally taken aback: it’s not fragile! Maybe Foolproof.
    Chelle Gentemann
    @cgentemann
    Fantastic?
    Fartastic?
    Rich Signell
    @rsignell-usgs
    Well with all the killed worker issues I've had recently, it was feeling pretty fragile
    but I guess it's really just user error
    Martin Durant
    @martindurant
    blame dask. Oh wait, that’s my responsibility (partly) too.
    Rich Signell
    @rsignell-usgs
    like this: pangeo-data/rechunker#100
    When the answer is "Set {"MALLOC_TRIM_THRESHOLD_": "0"} in the environment variables on your dask workers. " it feels fragile.
    Lucas Sterzinger
    @lsterzinger
    :laughing:
    Martin Durant
    @martindurant
    True, but nothing to do with reference-maker in this case. Of course, reference-maker has managed to crash things too.
    I actually wonder whether you
    ‘re just seeing transient memory usage during the processing of a task, due to temporary copies
    Chelle Gentemann
    @cgentemann
    Lucas - I just figured out access to the new MUR SST dataset that podaac threw up on the cloud. where is the best summary of what i need to do to create my .json file? still your medium article?
    Rich Signell
    @rsignell-usgs
    Chelle, here's my last notebook creating JSON for NetCDF files: https://nbviewer.jupyter.org/gist/rsignell-usgs/ef435a53ac530a2843ce7e1d59f96e22
    In case it's useful
    Chelle Gentemann
    @cgentemann
    :thumbsup:
    Lucas Sterzinger
    @lsterzinger
    @rsignell-usgs 's notebook is good, might also be worth checking out the first notebook in the workshop I'm giving today. Feel free to shoot me a message if you need help
    https://github.com/lsterzinger/maptimedavis-fsspec/blob/main/01-Create_References.ipynb
    Martin Durant
    @martindurant
    We need to collate these materials!
    (and the known reference’d datasets)
    Lucas Sterzinger
    @lsterzinger
    When I get a moment to breathe, I'm going to try and figure out why reference maker won't work with my research model's HDF5 output
    Because right now I spend the first few minutes of each analysis notebook building a mfdataset
    I also think it might be good to revamp the readme at fsspec-reference-maker, maybe link to a few examples/blog posts and give a quick overview of what it is, how it works, etc. Right now if I just stumbled upon the repo I wouldn't have a great idea of what it was all about
    Martin Durant
    @martindurant
    Agreed. A good think to do for hacktoberfest. I had in mind to turn to that only in time for my pydata talk.
    (end of the month)
    Martin Durant
    @martindurant
    I added hacktoberfest2021 to fsspec-reference-maker’s labels