Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Dave Hirschfeld
    @dhirschfeld

    [Stefan Krah] We could be fancy and check for the narrowest type among the input arguments and return that, but this sort of thing slows down small array operations.

    I'm pretty interested in keeping the overhead as minimal as possible so if that means having to specify the type (cls) then so be it however perhaps we can get the best of both worlds by skipping the type guessing heuristics in the case that cls is not None?

    [Pearu Peterson] Similarly for multiple argument cases, the first argument type defines the type of the output objects (default behavior).

    I'd be fine with this choice - so long as it's consistent it can be accomodated. I guess the other option would be return the type of the first common base but I assume that would have a much larger performance impact.

    Dave Hirschfeld
    @dhirschfeld
    As to whether or not sin should be a function or a method - this is an issue I see also in numpy in that the boundary for whats a method vs what's a function is somewhat blurred and opaque.
    For the most part I like the method-call version as it can be clearer if you're chaining several operations together
    I often get bitten by the fact that there is no abs method - only the function version
    I quite like pandas' concept of function namespaces - e.g. df.dt, df.str which hold functions for datetime and string operations respectively
    Dave Hirschfeld
    @dhirschfeld
    xnd.array could possibly use a similar concept where the xnd.array.ufunc "namespace" exposed the gumath.functions namespace with the first argument substituted for self
    Along with Pearu's suggestion that cls=type(args[0]) by default, this would ensure that the method-call versions returned the type of the instance
    Dave Hirschfeld
    @dhirschfeld
    e.g.
    class A(xnd): 
        pass
    
    class B(xnd):
        pass
    
    a = A([1])
    b = B([2])
    
    c = gumath.functions.add(a, b)
    assert type(c) == type(a)
    
    c = a.ufunc.add(b)
    assert type(c) == type(a)
    
    c = gumath.functions.add(b, a)
    assert type(c) == type(b)
    
    c = b.ufunc.add(a)
    assert type(c) == type(b)
    Dave Hirschfeld
    @dhirschfeld
    ...but cls can still be overridden:
    c = gumath.functions.add(a, b, cls=type(b))
    assert type(c) == type(b)
    
    c = a.ufunc.add(b, cls=type(b))
    assert type(c) == type(b)
    Anyway, just a thought. On the downside xnd.array.ufunc is a bit of a mouthful ¯_(ツ)_/¯
    Quansight
    @quansightai_twitter
    [Travis Oliphant] It’s true that you can’t just quote a principle nor is it always obvious whether inheritance or composition is a good idea. But, but, I wish I (or anyone else on the NumPy list at the time) had known about the Liskov substitution principle because it would have definitely led to np.matrix not being a sub-class of ndarray
    [Travis Oliphant] Because of the fundamental changes to the interface that were made (which were not merely additive). '
    [Travis Oliphant] Adding units (reasonable use of subclass)
    [Travis Oliphant] Masked arrays are a gray area for me. They can reasonably be subclasses.
    Quansight
    @quansightai_twitter
    [Hameer Abbasi] If we define Liskov in the following way: For a deterministic Operation on the parent class, data produced could have a child class, but the parent’s attributes and operations on that child should be identical to if the parent was passed in; or themselves satisfy Liskov.
    Quansight
    @quansightai_twitter

    [Hameer Abbasi] Then xnd.array is a valid subclass of xnd.xnd, but this doesn’t hold for MaskedArrays or matrix objects.

    astropy.Quantity is valid sometimes and sometimes errors, assuming consistency of units and no conversion. If inconsistent units are supplied, it violates Liskov in this form.

    If we have a much more relaxed definition, such that the dtype and the shape have to be consistent (type in xnd-speak), and errors are allowed, then MaskedArray and unit become valid.

    Quansight
    @quansightai_twitter
    [XND]
    Event starting in 15 minutes:
    XND meeting
    May 10th, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter
    [Travis Oliphant] Good illustration that Liskov alone does not resolve the question -- but it does provide useful scaffolding for the conversation that will help you make a decision you will be happy with later.
    Quansight
    @quansightai_twitter
    [XND] There is 1 event this week
    XND meeting
    May 17th, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter
    [XND]
    Calendar event was cancelled
    XND meeting
    May 17th, 2019 from 9:00 AM to 10:00 AM GMT-0400
    [Hameer Abbasi] @stefan.krah Is it possible to keep the meeting open by any chance? I had some productive discussion at BIDS I’d like everyone to hear.
    [Stefan Krah] I'd rather not, I also have another meeting tomorrow. Can it wait until next week?
    [Hameer Abbasi] Sure, if you prefer. I’ll try to make a brain dump so that it isn’t lost.
    Quansight
    @quansightai_twitter
    [Stefan Krah] Thanks, it will be interesting to hear about the discussion.
    Quansight
    @quansightai_twitter
    [XND] There is 1 event this week
    XND meeting
    May 24th, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter
    [XND]
    Calendar event was cancelled
    XND meeting
    May 24th, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter
    [XND] There is 1 event this week
    XND meeting
    May 31st, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter
    [Sameer Deshmukh] I'll moving all the ruby wrappers for the xnd project into this repo: https://github.com/xnd-project/xnd-ruby
    Quansight
    @quansightai_twitter

    [Sameer Deshmukh] @ralf.gommers I think this message states that the latest ndtypes does not need refcounting. Let me try updating this on the ruby side and then I can update you on what is going on as of now.
    An update on reference counted types and the history:

    The Numba team was concerned about having reference counted types in libndtypes. However, the more I looked at it, I can't see any harm. For example, ...

    t = ndt_string(ctx);
    ndt_del(t)

    .... is now just:

    t = ndt_string(ctx);
    ndt_decref(t);

    So refcounting is all internal to libndtypes and has nothing to do with Python refcounting.

    It is all implemented in the latest libndtypes.

    One implementation detail: An ndt_t * is now always const except for the refcnt field. ndt_decref(const ndt_t *) follows the convention in the Linux kernel, where kfree() takes a const void *.

    The convention in the Linux kernel has been the source of some controversy. I understand some of the counter arguments, but on the whole I think it's a good idea. More here:

    https://yarchive.net/comp/const.html

    Quansight
    @quansightai_twitter
    [Ralf Gommers] thanks @sameer.deshmukh. A separate xnd-ruby repo seems like a good idea to me. makes it visible, and it can be tested together with libxnd in CI easily.
    Quansight
    @quansightai_twitter
    [Hameer Abbasi] @sameer.deshmukh I think you may be wrong there.
    [Hameer Abbasi] It does ref count. It just says that a simple create and delete can be emulated by ref counting.
    Quansight
    @quansightai_twitter
    [Stefan Krah] Yes, there is ref-counting. That message is mostly concerned with the question in what way it could be harmful to JITs (I think it cannot) and to issue a preemptive rationalization for freeing a const *, which has Linus' endorsement but might lead to raised eyebrows elsewhere.
    Quansight
    @quansightai_twitter
    [XND] There is 1 event this week
    XND meeting
    June 7th, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter

    [Stefan Krah] I've implemented flexible ND arrays, which are more versatile than the offset-based var dimensions.

    These are needed for quirky formats like GeoSON.

    The trade-off is that they are not in a single memory block.

    You can now assign subarrays of different sizes:

    >>> 
    >>> x = xnd([[0], [1, 2]], type="array of array of int64") 
    >>> x
    xnd([[0], [1, 2]], type='array of array of int64')
    >>> x[0] = [1000, 2000, 3000]
    >>> x
    xnd([[1000, 2000, 3000], [1, 2]], type='array of array of int64')
    >>>
    Quansight
    @quansightai_twitter

    [Stefan Krah] Together with tagged unions one arrives at a far more condensed GeoJSON representation:

    `\

    Quansight
    @quansightai_twitter
    [Hameer Abbasi] This is great! So are these arrays refcounted or always copies?
    Quansight
    @quansightai_twitter
    [Stefan Krah] It's a copy. I wonder if things get too tricky with sharing. Not actually implementation-wise, but for the user.
    [Hameer Abbasi] Well, NumPy currently only does copies. But with XND, I would assume people know what they’re doing.
    Quansight
    @quansightai_twitter
    [Pearu Peterson] sharing should be fine when read-only
    Quansight
    @quansightai_twitter
    [XND] There is 1 event this week
    XND meeting
    June 14th, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter
    [XND]
    Calendar event was cancelled
    XND meeting
    June 14th, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter
    [XND] There is 1 event this week
    XND meeting
    June 21st, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter
    [Hameer Abbasi] I think it should
    Quansight
    @quansightai_twitter
    [XND]
    Event starting in 15 minutes:
    XND meeting
    June 21st, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Quansight
    @quansightai_twitter
    [Hameer Abbasi] xnd-dask should be possible for ndarray: dask/dask#4669

    [Stefan Krah] Nice, this also needs to be resolved, probably on the NumPy side:

    dask/dask#4632

    Quansight
    @quansightai_twitter
    [XND]
    Calendar event was cancelled
    XND meeting
    March 22nd, 2019 from 9:00 AM to 10:00 AM GMT-0400
    Recurring
    Quansight
    @quansightai_twitter
    [Stefan Krah] Meeting cancelled because of vacation time. Next is probably July 19th.