by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Martin Durant
    @martindurant
    I’ll be there on the hour
    Julia Signell
    @jsignell
    I think I've changed the zoom in all the places where it's written
    Martin Durant
    @martindurant
    We can reconvene on another zoom, if there’s more to day
    Thomas A Caswell
    @tacaswell
    I'm happy with where we left things :+1:
    Paul-Armand Verhaegen
    @paularmand
    at this point, I haven't yet managed to get my head around it completely, but I imagine an integration of intake with blazingsql on dask that returns dask-cudf? Is that on the radar ?
    Martin Durant
    @martindurant
    Sounds totally reasonable a thing to do, but I have no work in that direction
    Paul-Armand Verhaegen
    @paularmand
    ;) ... figured ;)
    I hear you.
    Martin Durant
    @martindurant
    If you don’t fancy working on it, feel free to poke blazingsql and ask
    Paul-Armand Verhaegen
    @paularmand
    If we go that way (blazingsql) than we'll contribute it back. I fancy that... but I'm an architect and I should be drawing boxes ;)
    Noushin
    @bnoushin7
    Hi, I have a question that might not be 100% related to intake though. I have NetCDF files in different directories which are named yearly like 1998 ,1999, 2000, 2001 to 2019. I want to catalog them together. I was able to use this code for 1998 and 1999. Please check this code:
    Noushin
    @bnoushin7
    image.png
    Noushin
    @bnoushin7
    How can I add 2000-2019 to this command?
    I tried path='/shared/land/GEWEX_IP/V_2019/199[8-9]|20[01-19]/' but it is not even interpreted
    Martin Durant
    @martindurant
    Indeed, I don’t think that’s legal syntax. Unfortunately, paths are interpreted as shell globs, whereas I suppose you could have expressed this in direct regex. There is also no way to pass a set of globs, it’s either a single strign that gets expanded, or a list of files. If it impractical to provide a list of files? You could propose in an issu to fsspec, that a list of globs should be expanded one-by-one to an accumulated set of paths in this kind of operation.
    Noushin
    @bnoushin7
    @martindurant Thank you
    Thomas A Caswell
    @tacaswell
    Did the master branch get force-pushed to?
    Martin Durant
    @martindurant
    not as far as I know - what’s up?
    Thomas A Caswell
    @tacaswell
    4e7a44b9d29d1ecd8f88fb567165f9f99b30c0a8 is where my local master branch was sitting
    but is no longer in the history of current master
    but I apparently pushed it to my fork
    Martin Durant
    @martindurant
    Seems to be the same as intake/intake@6bf4a18 - a squashed version
    Thomas A Caswell
    @tacaswell
    Hmm, I suspect I use the gh cli and get my self confused
    sorry for the noise
    Martin Durant
    @martindurant
    No need to be sorry! Are you working on intake?
    Thomas A Caswell
    @tacaswell
    my re-build-the-world script failed because it could to fast-forward pull intake
    Martin Durant
    @martindurant
    I’m really glad I don’t have a script like that.
    Any idea when you might be working on the splitting of DataSource into separate functionalality classes?
    Thomas A Caswell
    @tacaswell
    maybe next week. I have been well behind on the time I own MPL vs the time I owe BNL
    Noushin
    @bnoushin7
    Hi @martindurant I know this is an array problem, but I was thinking maybe you have a clue. @bnoushin7
    Hi, I have a problem with xarray.opendataset. I have a file containing the address of the dataset. When I read it in the bash and want to pass it to the opendataset, it gives me an error that file does not exist. And that is because it concat the address of the current directory to the file address that I am reading from the file. Any clue?
    Here is what I get:
    why does it add b/homes/nbehboud/test which is the current directory to the real file path?
    KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/homes/nbehboud/test/"/shared/obs/gridded/GHCN_CAMS/air.mon.mean.nc"',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]
    Thank you
    Martin Durant
    @martindurant
    Does this happen when you open the file directly with xarray, as oposed to with intake?
    tomerten
    @tomerten
    Hi everybody, I'm new to intake and I am struggling with authentication/visibility of sources in a catalog. Is there a way to set which source in a catalog can be viewed by who and password protect them, the documentation confused me and I could not find a clear example in the examples provided. My catalog is on a remote intake-server behind a nginx reverse proxy with authentication (still working on the TSL/SSL part). Thanks.
    Martin Durant
    @martindurant
    The standard "shared secret” example shows you how to grant access on the basis of a header keyword. In your case, I suppose the auth-proxy populates the header with the user’s ID, so the change would be to take this value and make a decision based on it. Somewhere, you will have to figure out how to store/load which user should see which entries.
    tomerten
    @tomerten
    Hi Martin, thanks for your reply. But I still do not see how for example (forget about the nginx for the moment) if I have two csv files and two users how I can make only one file visible to one of the users while the other one can see both. More specifically, can you show me an example of a catalog yaml file where this is done?
    Martin Durant
    @martindurant
    If you post this to stackoverflow, I can provide an example. It’s better to answer there, so that others can find it too - please use the [intake] tag
    tomerten
    @tomerten
    Hi Martin, posted it on stack overflow (https:stackoverflow.com/questions/62590078/dataset-view-and-access-control-in-yaml-file).
    Martin Durant
    @martindurant
    I’ll answer when I get a little free time
    Dan Allan
    @danielballan
    I'm taking a long weekend around July 4 so I'll miss this month's intake dev meeting. Next issue in my sights is intake/intake#506, and then helping @tacaswell with refactoring the base classes into mixins.
    Martin Durant
    @martindurant
    Actually, I was thinking to take time off, bridging Canada Day to the weekend. Postpone for one week?
    Dan Allan
    @danielballan
    Sounds good to me!
    tomerten
    @tomerten
    @martindurant Hi I posted another question on stack overflow regarding customised return formats, similar to 'to_dask()'. Thanks for your previous answer, that was very useful.
    Martin Durant
    @martindurant
    I’ll get back to you @tomerten
    Dan Allan
    @danielballan
    It occurred to me that postponing one week would put us up against SciPy. Postpone two weeks?
    Martin Durant
    @martindurant
    Ugh. Lets have a quick catchup anyway. We can always have another meeting.
    (wonders if he’s supposed to be doing something for SciPy…)
    Dan Allan
    @danielballan
    That sounds fine with me.
    Martin Durant
    @martindurant
    Are you doing anything scipyish?
    Dan Allan
    @danielballan
    Mentoring someone, but that's all.