Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Rich Signell
    @rsignell-usgs
    Lucas, to me it's important you don't say that you need to read the whole netcdf4 file to get the metadata.
    And also note that the compressed chunks of data in NetCDF4 can be extracted just as cloud-friendly as Zarr format
    Lucas Sterzinger
    @lsterzinger

    Lucas, to me it's important you don't say that you need to read the whole netcdf4 file to get the metadata

    Yes I agree. This is a piece of information I got from a conversation with Kevin Paul at NCAR -- I think I either misunderstood what he was saying (the most likely reason) or he misunderstood exactly how cloud access happens... maybe thinking of a time where you could only request entire files from object storage.

    I will make updates to the presentation before the workshop, thank you for your help!

    Rich Signell
    @rsignell-usgs
    :thumbsup:
    Good luck with the workshop! May your clusters spawn!
    I was joking with Zac Flamig we might call our framework "Fragile Architecture for Reproducible and Transparent Science"
    Martin Durant
    @martindurant
    Farts?
    Rich Signell
    @rsignell-usgs
    :thumbsup:
    I'm about to turn 60 and still making fart jokes. That's good, right?
    :)
    Martin Durant
    @martindurant
    It never gets old
    By the way @rsignell-usgs , you mentioned possible funding when I was talking about extra viz tools for our kind of big datasets, e.g., to know which parts of the satellite imagery contain data. JimB has a person that might have hours available for that kind of thing.
    Chelle Gentemann
    @cgentemann
    I wake up to FARTS. love you all. :)
    Rich Signell
    @rsignell-usgs
    @martindurant , okay, I'll see what I can do
    @cgentemann , phew! (or should I say "piu"!)
    Martin Durant
    @martindurant
    I was initally taken aback: it’s not fragile! Maybe Foolproof.
    Chelle Gentemann
    @cgentemann
    Fantastic?
    Fartastic?
    Rich Signell
    @rsignell-usgs
    Well with all the killed worker issues I've had recently, it was feeling pretty fragile
    but I guess it's really just user error
    Martin Durant
    @martindurant
    blame dask. Oh wait, that’s my responsibility (partly) too.
    Rich Signell
    @rsignell-usgs
    like this: pangeo-data/rechunker#100
    When the answer is "Set {"MALLOC_TRIM_THRESHOLD_": "0"} in the environment variables on your dask workers. " it feels fragile.
    Lucas Sterzinger
    @lsterzinger
    :laughing:
    Martin Durant
    @martindurant
    True, but nothing to do with reference-maker in this case. Of course, reference-maker has managed to crash things too.
    I actually wonder whether you
    ‘re just seeing transient memory usage during the processing of a task, due to temporary copies
    Chelle Gentemann
    @cgentemann
    Lucas - I just figured out access to the new MUR SST dataset that podaac threw up on the cloud. where is the best summary of what i need to do to create my .json file? still your medium article?
    Rich Signell
    @rsignell-usgs
    Chelle, here's my last notebook creating JSON for NetCDF files: https://nbviewer.jupyter.org/gist/rsignell-usgs/ef435a53ac530a2843ce7e1d59f96e22
    In case it's useful
    Chelle Gentemann
    @cgentemann
    :thumbsup:
    Lucas Sterzinger
    @lsterzinger
    @rsignell-usgs 's notebook is good, might also be worth checking out the first notebook in the workshop I'm giving today. Feel free to shoot me a message if you need help
    https://github.com/lsterzinger/maptimedavis-fsspec/blob/main/01-Create_References.ipynb
    Martin Durant
    @martindurant
    We need to collate these materials!
    (and the known reference’d datasets)
    Lucas Sterzinger
    @lsterzinger
    When I get a moment to breathe, I'm going to try and figure out why reference maker won't work with my research model's HDF5 output
    Because right now I spend the first few minutes of each analysis notebook building a mfdataset
    I also think it might be good to revamp the readme at fsspec-reference-maker, maybe link to a few examples/blog posts and give a quick overview of what it is, how it works, etc. Right now if I just stumbled upon the repo I wouldn't have a great idea of what it was all about
    Martin Durant
    @martindurant
    Agreed. A good think to do for hacktoberfest. I had in mind to turn to that only in time for my pydata talk.
    (end of the month)
    Martin Durant
    @martindurant
    I added hacktoberfest2021 to fsspec-reference-maker’s labels
    Lucas Sterzinger
    @lsterzinger
    I think it's supposed to just be hacktoberfest, but I'm not sure. That's what I tagged one of my repos as and I got credit for merging a PR on it
    Martin Durant
    @martindurant
    Added both :|
    Rich Signell
    @rsignell-usgs
    Chelle, is this the data you want to create JSON for? https://github.com/mgangl/tds-mur-test
    I'm on the netcdf-java-mailing list, and NASA is going crazy with slow access speeds to these NetCDF files
    Chelle Gentemann
    @cgentemann
    yes - that is exactly what i'm working on right now
    i have your code up & am all connected.
    just working on writing json files
    can you send me some of the chatter on slow access?
    Rich Signell
    @rsignell-usgs
    Sure