Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Lucas Sterzinger
    @lsterzinger
    When I get a moment to breathe, I'm going to try and figure out why reference maker won't work with my research model's HDF5 output
    Because right now I spend the first few minutes of each analysis notebook building a mfdataset
    I also think it might be good to revamp the readme at fsspec-reference-maker, maybe link to a few examples/blog posts and give a quick overview of what it is, how it works, etc. Right now if I just stumbled upon the repo I wouldn't have a great idea of what it was all about
    Martin Durant
    @martindurant
    Agreed. A good think to do for hacktoberfest. I had in mind to turn to that only in time for my pydata talk.
    (end of the month)
    Martin Durant
    @martindurant
    I added hacktoberfest2021 to fsspec-reference-maker’s labels
    Lucas Sterzinger
    @lsterzinger
    I think it's supposed to just be hacktoberfest, but I'm not sure. That's what I tagged one of my repos as and I got credit for merging a PR on it
    Martin Durant
    @martindurant
    Added both :|
    Rich Signell
    @rsignell-usgs
    Chelle, is this the data you want to create JSON for? https://github.com/mgangl/tds-mur-test
    I'm on the netcdf-java-mailing list, and NASA is going crazy with slow access speeds to these NetCDF files
    Chelle Gentemann
    @cgentemann
    yes - that is exactly what i'm working on right now
    i have your code up & am all connected.
    just working on writing json files
    can you send me some of the chatter on slow access?
    Rich Signell
    @rsignell-usgs
    Sure
    Chelle Gentemann
    @cgentemann
    holy crap maybe i'm actually getting it to work. fingers crossed!
    Rich Signell
    @rsignell-usgs
    go Chelle go!
    Chelle Gentemann
    @cgentemann
    i'm so at the very edge of my understanding of file systems. someday i need martin to do a vulcan mind meld with me so it all is clear.
    Martin Durant
    @martindurant
    Actually, disseminating this information is supposed to be part of my job. Along with most of my pure-code brethren, I’m not particularly good at writing and structuring documentation in a way that people can find it. Actually, that tends to be true of academic researchers too.
    Rich Signell
    @rsignell-usgs
    Perhaps you, Lucas and I all should meet with Martin, and we could try giving our best explanation of what we think we know, and then Martin can upgrade our understanding
    And then we would be in a better position to help with the docs
    and get our t-shirts
    Chelle Gentemann
    @cgentemann
    i think it would be like one of those funny youtube videos where adults ask children to explain concepts and their idea of what it is is so far from reality it ends up being funny. ;)
    Rich Signell
    @rsignell-usgs
    Yeah, exactly
    Martin Durant
    @martindurant
    Or, that the child can explain it far simpler than the adult.
    Rich Signell
    @rsignell-usgs
    When my daughter was 11 she suggested we should make her backpack lighter by filling it with "that air they have on the moon"
    Martin Durant
    @martindurant
    Teh README at /tds-mur-test is pretty unenlightening
    Chelle Gentemann
    @cgentemann
    last night austin asked if we could fill a balloon with the something lighter than helium and make our prius into a flying car.
    Martin Durant
    @martindurant
    Yes you could! Might be a big balloon.
    Rich Signell
    @rsignell-usgs
    So Martin, yes, we need your help in understanding, in other words
    Martin Durant
    @martindurant
    got it
    Chelle Gentemann
    @cgentemann
    i'm just gonna create a channel where my kids can ask martin questions instead of me.
    Rich Signell
    @rsignell-usgs
    Also reading that data seems to require an AWS profile I don't have
    Chelle Gentemann
    @cgentemann
    i've got that all worked out
    i've made json for 30 files, working on putting together before i run on bigger set
    Rich Signell
    @rsignell-usgs
    cool. Chelle, when you get it, let's do a screeshare
    Chelle Gentemann
    @cgentemann
    kk
    Rich Signell
    @rsignell-usgs
    Probably goes without saying, but if you have a lot of files to create individual jsons for, a bigger dask cluster helps.
    Lucas Sterzinger
    @lsterzinger
    I'm sure time will beat this out of me, but I don't mind writing documentation/tutorials/examples (obviously, by my contributions to reference maker thusfar)
    The workshop went well, managed to crash pangeo binder for about 10-15 minutes so I guess it's time to scratch another notch into the side of my computer
    Chelle Gentemann
    @cgentemann
    glad to hear it went well! Rich - it is totally working. I feel like a real hacker now! ;) i'm free the rest of the day - when would be good - i'd like to go over it with you a little bit before scaling up.
    Rich Signell
    @rsignell-usgs
    Martin, I told Chelle I knew how to get the NASA credentials to Dask workers, but now that I look at it, I'm not sure I do. Can you look at cell [3] here and recommend a path?
    https://github.com/cgentemann/cloud_science/blob/master/make_zarr/cloud_mur_v41.ipynb
    I thought we could just copy the .netrc to workers using a dask.distributed WorkerPlugin, but I'm not sure that would do it (or is even what we would want to do)
    Lucas Sterzinger
    @lsterzinger
    Can you pass environment variables to the workers?
    Rich Signell
    @rsignell-usgs
    I'm guessing we might need to run the begin_s3_direct_access script on all the workers via a WorkerPlugin.
    Martin Durant
    @martindurant
    Bahh, I just had a conversation exactly on this sort of thing, for GCP. Looking...
    Since you are using an s3fs instance with explicit token values, it should go to the workers just fine, without having to reestablish credentials
    Rich Signell
    @rsignell-usgs
    And of course I can use fsspec instead of s3fs
    right?
    Martin Durant
    @martindurant
    fsspec.filesystem(“s3”) is identical to s3fs.S3FileSystem(), but perhaps better aesthetically