Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Martin Durant
    @martindurant
    It never gets old
    By the way @rsignell-usgs , you mentioned possible funding when I was talking about extra viz tools for our kind of big datasets, e.g., to know which parts of the satellite imagery contain data. JimB has a person that might have hours available for that kind of thing.
    Chelle Gentemann
    @cgentemann
    I wake up to FARTS. love you all. :)
    Rich Signell
    @rsignell-usgs
    @martindurant , okay, I'll see what I can do
    @cgentemann , phew! (or should I say "piu"!)
    Martin Durant
    @martindurant
    I was initally taken aback: it’s not fragile! Maybe Foolproof.
    Chelle Gentemann
    @cgentemann
    Fantastic?
    Fartastic?
    Rich Signell
    @rsignell-usgs
    Well with all the killed worker issues I've had recently, it was feeling pretty fragile
    but I guess it's really just user error
    Martin Durant
    @martindurant
    blame dask. Oh wait, that’s my responsibility (partly) too.
    Rich Signell
    @rsignell-usgs
    When the answer is "Set {"MALLOC_TRIM_THRESHOLD_": "0"} in the environment variables on your dask workers. " it feels fragile.
    Lucas Sterzinger
    @lsterzinger
    :laughing:
    Martin Durant
    @martindurant
    True, but nothing to do with reference-maker in this case. Of course, reference-maker has managed to crash things too.
    I actually wonder whether you
    ‘re just seeing transient memory usage during the processing of a task, due to temporary copies
    Chelle Gentemann
    @cgentemann
    Lucas - I just figured out access to the new MUR SST dataset that podaac threw up on the cloud. where is the best summary of what i need to do to create my .json file? still your medium article?
    Rich Signell
    @rsignell-usgs
    Chelle, here's my last notebook creating JSON for NetCDF files: https://nbviewer.jupyter.org/gist/rsignell-usgs/ef435a53ac530a2843ce7e1d59f96e22
    In case it's useful
    Chelle Gentemann
    @cgentemann
    :thumbsup:
    Lucas Sterzinger
    @lsterzinger
    @rsignell-usgs 's notebook is good, might also be worth checking out the first notebook in the workshop I'm giving today. Feel free to shoot me a message if you need help
    Martin Durant
    @martindurant
    We need to collate these materials!
    (and the known reference’d datasets)
    Lucas Sterzinger
    @lsterzinger
    When I get a moment to breathe, I'm going to try and figure out why reference maker won't work with my research model's HDF5 output
    Because right now I spend the first few minutes of each analysis notebook building a mfdataset
    I also think it might be good to revamp the readme at fsspec-reference-maker, maybe link to a few examples/blog posts and give a quick overview of what it is, how it works, etc. Right now if I just stumbled upon the repo I wouldn't have a great idea of what it was all about
    Martin Durant
    @martindurant
    Agreed. A good think to do for hacktoberfest. I had in mind to turn to that only in time for my pydata talk.
    (end of the month)
    Martin Durant
    @martindurant
    I added hacktoberfest2021 to fsspec-reference-maker’s labels
    Lucas Sterzinger
    @lsterzinger
    I think it's supposed to just be hacktoberfest, but I'm not sure. That's what I tagged one of my repos as and I got credit for merging a PR on it
    Martin Durant
    @martindurant
    Added both :|
    Rich Signell
    @rsignell-usgs
    Chelle, is this the data you want to create JSON for? https://github.com/mgangl/tds-mur-test
    I'm on the netcdf-java-mailing list, and NASA is going crazy with slow access speeds to these NetCDF files
    Chelle Gentemann
    @cgentemann
    yes - that is exactly what i'm working on right now
    i have your code up & am all connected.
    just working on writing json files
    can you send me some of the chatter on slow access?
    Rich Signell
    @rsignell-usgs
    Sure
    Chelle Gentemann
    @cgentemann
    holy crap maybe i'm actually getting it to work. fingers crossed!
    Rich Signell
    @rsignell-usgs
    go Chelle go!
    Chelle Gentemann
    @cgentemann
    i'm so at the very edge of my understanding of file systems. someday i need martin to do a vulcan mind meld with me so it all is clear.
    Martin Durant
    @martindurant
    Actually, disseminating this information is supposed to be part of my job. Along with most of my pure-code brethren, I’m not particularly good at writing and structuring documentation in a way that people can find it. Actually, that tends to be true of academic researchers too.
    Rich Signell
    @rsignell-usgs
    Perhaps you, Lucas and I all should meet with Martin, and we could try giving our best explanation of what we think we know, and then Martin can upgrade our understanding
    And then we would be in a better position to help with the docs
    and get our t-shirts
    Chelle Gentemann
    @cgentemann
    i think it would be like one of those funny youtube videos where adults ask children to explain concepts and their idea of what it is is so far from reality it ends up being funny. ;)
    Rich Signell
    @rsignell-usgs
    Yeah, exactly
    Martin Durant
    @martindurant
    Or, that the child can explain it far simpler than the adult.