Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Rich Signell
    @rsignell-usgs
    :thumbsup:
    Good luck with the workshop! May your clusters spawn!
    I was joking with Zac Flamig we might call our framework "Fragile Architecture for Reproducible and Transparent Science"
    Martin Durant
    @martindurant
    Farts?
    Rich Signell
    @rsignell-usgs
    :thumbsup:
    I'm about to turn 60 and still making fart jokes. That's good, right?
    :)
    Martin Durant
    @martindurant
    It never gets old
    By the way @rsignell-usgs , you mentioned possible funding when I was talking about extra viz tools for our kind of big datasets, e.g., to know which parts of the satellite imagery contain data. JimB has a person that might have hours available for that kind of thing.
    Chelle Gentemann
    @cgentemann
    I wake up to FARTS. love you all. :)
    Rich Signell
    @rsignell-usgs
    @martindurant , okay, I'll see what I can do
    @cgentemann , phew! (or should I say "piu"!)
    Martin Durant
    @martindurant
    I was initally taken aback: it’s not fragile! Maybe Foolproof.
    Chelle Gentemann
    @cgentemann
    Fantastic?
    Fartastic?
    Rich Signell
    @rsignell-usgs
    Well with all the killed worker issues I've had recently, it was feeling pretty fragile
    but I guess it's really just user error
    Martin Durant
    @martindurant
    blame dask. Oh wait, that’s my responsibility (partly) too.
    Rich Signell
    @rsignell-usgs
    like this: pangeo-data/rechunker#100
    When the answer is "Set {"MALLOC_TRIM_THRESHOLD_": "0"} in the environment variables on your dask workers. " it feels fragile.
    Lucas Sterzinger
    @lsterzinger
    :laughing:
    Martin Durant
    @martindurant
    True, but nothing to do with reference-maker in this case. Of course, reference-maker has managed to crash things too.
    I actually wonder whether you
    ‘re just seeing transient memory usage during the processing of a task, due to temporary copies
    Chelle Gentemann
    @cgentemann
    Lucas - I just figured out access to the new MUR SST dataset that podaac threw up on the cloud. where is the best summary of what i need to do to create my .json file? still your medium article?
    Rich Signell
    @rsignell-usgs
    Chelle, here's my last notebook creating JSON for NetCDF files: https://nbviewer.jupyter.org/gist/rsignell-usgs/ef435a53ac530a2843ce7e1d59f96e22
    In case it's useful
    Chelle Gentemann
    @cgentemann
    :thumbsup:
    Lucas Sterzinger
    @lsterzinger
    @rsignell-usgs 's notebook is good, might also be worth checking out the first notebook in the workshop I'm giving today. Feel free to shoot me a message if you need help
    https://github.com/lsterzinger/maptimedavis-fsspec/blob/main/01-Create_References.ipynb
    Martin Durant
    @martindurant
    We need to collate these materials!
    (and the known reference’d datasets)
    Lucas Sterzinger
    @lsterzinger
    When I get a moment to breathe, I'm going to try and figure out why reference maker won't work with my research model's HDF5 output
    Because right now I spend the first few minutes of each analysis notebook building a mfdataset
    I also think it might be good to revamp the readme at fsspec-reference-maker, maybe link to a few examples/blog posts and give a quick overview of what it is, how it works, etc. Right now if I just stumbled upon the repo I wouldn't have a great idea of what it was all about
    Martin Durant
    @martindurant
    Agreed. A good think to do for hacktoberfest. I had in mind to turn to that only in time for my pydata talk.
    (end of the month)
    Martin Durant
    @martindurant
    I added hacktoberfest2021 to fsspec-reference-maker’s labels
    Lucas Sterzinger
    @lsterzinger
    I think it's supposed to just be hacktoberfest, but I'm not sure. That's what I tagged one of my repos as and I got credit for merging a PR on it
    Martin Durant
    @martindurant
    Added both :|
    Rich Signell
    @rsignell-usgs
    Chelle, is this the data you want to create JSON for? https://github.com/mgangl/tds-mur-test
    I'm on the netcdf-java-mailing list, and NASA is going crazy with slow access speeds to these NetCDF files
    Chelle Gentemann
    @cgentemann
    yes - that is exactly what i'm working on right now
    i have your code up & am all connected.
    just working on writing json files
    can you send me some of the chatter on slow access?
    Rich Signell
    @rsignell-usgs
    Sure
    Chelle Gentemann
    @cgentemann
    holy crap maybe i'm actually getting it to work. fingers crossed!
    Rich Signell
    @rsignell-usgs
    go Chelle go!
    Chelle Gentemann
    @cgentemann
    i'm so at the very edge of my understanding of file systems. someday i need martin to do a vulcan mind meld with me so it all is clear.