Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Quansight
    @quansightai_twitter
    [Saul Shanabrook] FYI Pearu and I are working on numba /xnd/gumath integration in a new repo https://github.com/Quansight/numba-xnd. Once we have things settled down, we can try to add to numba core.
    Quansight
    @quansightai_twitter
    [Travis Oliphant] Great. I think that is a fine approach.
    Quansight
    @quansightai_twitter
    [Saul Shanabrook] @pearu.peterson I was able to switch the ndtypes/libndtypes numba integration to use xnd_structinfo and remove that sizes workaround I had. I wasn't able to switch the structs to be opaque pointers in Numba, because I actually have to be able to allocate empty ndt_array_t and ndt_context_t to pass into ndt_as_ndarray.
    Quansight
    @quansightai_twitter
    [Saul Shanabrook] Next for me is replicating ndtypes/libndtypes folders for xnd. I was able to remove using the pycapsule approach for unboxing, but to remove it for boxing we would have to figure out how to allocate a Ndt pyobject, like in https://github.com/plures/ndtypes/blob/6a6b0287fb8286820fa5c624be5a2fde81133ffc/python/ndtypes/_ndtypes.c#L1100-L1113. So I guess we would have to call the tp_alloc function on the Ndt_Type struct?
    Quansight
    @quansightai_twitter

    [Pearu Peterson] @saul, I believe that for creating Ndt pyobject we can use xnd_structinfo approach by implementing ndtype_alloc in LLVM IR. That was my initial thought and would involve getting pointer to Ndt_Type value, adding PyTypeObject struct wrapper to xnd_structinfo and implementing setter methods within xnd_structinfo. All these steps are doable.

    However, we can implement the creation of a Ndt pyobject witin a C function and then expose this function to LLVM IR. It can be done either within _ndtypes.c by defining the function as extern (easy approach in short run), or (in order to keep ndtypes clean from numba related stuff) creating a dummy-python extension module that implements the required extern function (it would be just harder to get a reference to Ndt_Type). In either case the extern function is made available to LLVM IR by importing the python module and binding it as shared library to LLVM as we currently do.

    If the approach works, we might need the dummy-python extension module also for other tasks that are easier to do in C (have direct access to struct members) than in LLVM IR (when pointer passing is not sufficient). Such functions can be added to xnd_structinfo, for instance, in order to keep the number of extension modules minimal.

    Quansight
    @quansightai_twitter
    [Pearu Peterson] @saul, I started to implement the approach and it turned out very simple:
    1. added extern PyObject* ndt_from_type(ndt_t* val) { return Ndt_FromType(val); } to xnd_infostruct.c
    2. run python setup.py develop
    Quansight
    @quansightai_twitter
    [Saul Shanabrook] Great!
    [Pearu Peterson] I'll push the code in a few minutes.
    [Saul Shanabrook] IMG_20180712_134331.jpg
    Quansight
    @quansightai_twitter
    [Pearu Peterson] The code is pushed in. Few notes:
    1. to add other C functions that can be used from LLVM IR, these must be added to structinfo_config.c_source string.
    2. I didn't remove pycapsule related code but I believe this can be cleaned out
    Quansight
    @quansightai_twitter

    [Saul Shanabrook] I have enough lower level primitives built up now in the numba support to jit compile a matrix multiply function: https://github.com/Quansight/numba-xnd/blob/8cdd89362aedde4516afedc44b21c5fe7aa9bf3c/tests/test_examples.py

    As you can see, it's very low level, similar to how you would write it in C, but in Python. Now I can start building the higher level APIs on top of that that have some more compile time typing knowledge.

    [Ivan Ogasawara] very nice!
    Quansight
    @quansightai_twitter
    [Saul Shanabrook] @anderson.banihirwe Do you think https://xndframes.readthedocs.io/en/latest/index.html is usable enough to add a link to it from xnd.io?
    Quansight
    @quansightai_twitter
    [Anderson] @saul, not yet... I still need to do extensive testing to make sure that the pandas API integration works as expected. Once this is taken care of, we can add it to xnd.io earlier next week hopefully.
    Raul Maldonado
    @Cloud_Chaoszero_twitter
    Hello, I was wondering what are the latest efforts of embedding or using ML with XND. (e.g. use XND within sci-kit learn)
    Quansight
    @quansightai_twitter
    [Travis Oliphant] We are still a ways off from that. The recent uarray efforts in Quansight-Labs are about creating an interface that could enable scikit-learn to use xnd
    [Travis Oliphant] We need to create a generic array object and a multiple-dispatch mechanism that can seamlessly replace scikit-learn usage of NumPy
    [Travis Oliphant] Best guess is next year if funding for Quansight Labs proceeds
    [Travis Oliphant] More in the immediate future would be looking to see how projects like dask could use xnd (and then dask-ml)
    Travis E. Oliphant
    @teoliphant
    Hey all. A good way forward, I think would be to look at how xnd could be used to live underneath the new C++ tensors PyTorch uses as well as the C++ library that Chainer uses.
    Who would be willing to look at that.
    Also, someone could look at how XND could be used to augment the type-system of Tensorflow.
    I suspect Chainer might be the easiest, but Torch and Tensorflow the most useful long-term.
    Quansight
    @quansightai_twitter
    [Travis Oliphant] Another really good use-case for xnd is xarray --- you could have an xnd backend to xarray.
    Quansight
    @quansightai_twitter
    [Travis Oliphant] One way to get gufunctions is to use things like RAPIDS and other C++ libraries to build the functionality
    [Travis Oliphant] We just need to wrap it with the gufunc concept.
    [Travis Oliphant] I was talking to Frederic Bastien and he was commenting how C++ compilers are much better now and C++ makes it easier to write low-level code well.
    [Travis Oliphant] XND should not limit gufunc implementations --- but only require a C-ABI to call the resulting code.
    [Travis Oliphant] RAPIDS is producing a lot of good c++ code that can be turned into XND gufuncs
    [Stefan Krah] XND uses C++ CUDA kernels.
    Quansight
    @quansightai_twitter
    [Travis Oliphant] We should put an XND interface to RAPIDS
    [Stefan Krah] So as long as the kernels have headers with extern "C" for the C++ functions, all is well.
    [Travis Oliphant] Yeah, that’s what I was thinking.
    [Stefan Krah] I'll look at RAPIDS, it sounds interesting.
    [Pearu Peterson] RAPIDS cuml seems to use C++ interface
    Quansight
    @quansightai_twitter
    [Travis Oliphant] https://github.com/RAPIDSai
    [Travis Oliphant] This is a great use-case for gumath --- to provide RAPIDS functionality to xnd containers
    [Travis Oliphant] if all of these are gumath functions, we can add these to uscipy
    [Travis Oliphant] with the xndarray backend.
    [Stefan Krah] I'm essentially using e.g. thrust::complex (also C++ obviously) by writing C wrapper macros.
    [Stefan Krah] So I guess the same would be possible for RAPIDS.
    Quansight
    @quansightai_twitter
    [Travis Oliphant] Yes.
    [Travis Oliphant] And very valuable because RAPIDS is getting a lot of attention and 2) we are going to be teaching RAPIDS and can start adding xnd to our teaching
    [Stefan Krah] That sounds great.
    [Travis Oliphant] I think adding RAPIDS as gufuncs callable by gumath would give us demoable work that we can leverage today
    [Travis Oliphant] and get people interested in xnd --- including potential RAPIDS people.
    [Travis Oliphant] @stefan.krah if you could wrap a couple of these functions (maybe one from cuML and one from cuGraph) it would help show the way that others can follow-
    Quansight
    @quansightai_twitter
    [Stefan Krah] Ok, I can do that.
    Quansight
    @quansightai_twitter
    [Travis Oliphant] We can show off these interfaces to RAPIDS at the SciPy tutorial Anthony is giving