Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 16:12
    github-actions[bot] labeled #3051
  • 16:11

    henryiii on master

    fix(clang-tidy): performance fi… (compare)

  • 16:11
    henryiii closed #3051
  • 15:07
    omer-keskin edited #3054
  • 15:01
    Skylion007 synchronize #3051
  • 14:59
    Skylion007 synchronize #3051
  • 14:56
    omer-keskin edited #3054
  • 14:49
    omer-keskin edited #3054
  • 14:49
    Skylion007 synchronize #3051
  • 14:39
    omer-keskin edited #3054
  • 14:30
    omer-keskin edited #3054
  • 14:26
    omer-keskin opened #3054
  • 14:11
    rfeinman commented #3035
  • 01:12
    rwgk synchronize #3023
  • Jun 21 23:17
    rwgk unlabeled #3053
  • Jun 21 23:16
    rwgk synchronize #3023
  • Jun 21 19:40
    github-actions[bot] labeled #3053
  • Jun 21 19:40

    rwgk on smart_holder

    [smart_holder] clang-tidy fixes… (compare)

  • Jun 21 19:40
    rwgk closed #3053
  • Jun 21 19:40
    rwgk commented #3053
Ali K
@alikesu_twitter

Hi, is it possible to use a py::dict as an instance variable when embedding a class which calls py::initialize_interpreter() in its constructor and py::finalize_interpreter() in its destructor?

The following case is crashing on me:

#include <pybind11/embed.h>

namespace py = pybind11;

class Node 
{
private:
    py::dict adict;

public:
    Node() {        
        py::initialize_interpreter();
    }

    ~Node() {
        py::finalize_interpreter();
    }
}
15 replies
Paul Gessinger
@paulgessinger
hey! i’m trying to use pybind11 to create bindings for a struct whose members are std::function. I want to be able to assign a C++ object which is callable, without the call going through python. When I simply assign a C++ object (whose type pybind11 is aware of) which has an operator(), it fails to resolve and gives an error. When I manually augment the object with a __call__ operator via pybind11 it resolves successfully. Now the call goes through python, however, which I’d like to avoid. Is there any way around this?
11 replies
Paul Gessinger
@paulgessinger
Ah I think the Callable[] is the python type assigned to std:function
quantotto
@quantotto
yepp
Paul Gessinger
@paulgessinger
Ok i See, but this is not using a callable struct an operator() anymore
I can work around the issue by modifying the C++ setup (that’s what I’ve done now) but I was wondering if it’s possible to have pybind11 correctly convert the struct without round tripping the call.
quantotto
@quantotto
I think the issue with the original code is that it can't implicitly convert your struct Callable to function pointer. Maybe you could create a base class, say BaseCallable that has operator() and then derive from it and implement it in derived classes as needed.
In the Outer, you would have BaseCallable func. Then you could assign objects derived from BaseCallable (thought it could require some trampoline classes to assure derived are accepted by Python). I think it is doable, but more complex and it is all a function of how important it is.
BTW, when you say round-tripping, what do you mean? Is it a call from C++, that goes to Python and then back to C++? If the call is from Python, it goes through Python anyways (Python -> C++) or am I missing something?
Paul Gessinger
@paulgessinger
If I manually add a call operator to the binding then it goes C++ -> Python-> C++. I noticed this because I get deadlocks when this function is called from multiple threads that were spawned in C++ because the python layer in between holds tries to get the GIL.
1 reply
katrina-petroske
@katrina-petroske

Can someone give me or point me to a simple example (I am a novice at both C++ and python), that includes the C++ and pybind code, for when you have a C++ function that outputs an array and you want the python wrapping to output an array as well?
What I have in C++ is:
Void functionName(blah, blah, array)
{
Modify array
{

Int main()
{
Double array[]
functionName(blah, blah, array)
}

What I want in python is something that runs like:
Import wrappedFunction

Result = wrappedFunction(whatever_Is_needed)

1 reply
nikhilTkur
@nikhilTkur

I have a few basic questions regarding pybind:
1) How do I bind the dict{int , dict{int , int}} to and unordered_map<int , unordered_map<int,int>>without copying
2) how to bind dict{int , list[tuples]} to unordered_map<int , std::vector<tuples>> without copying

Can someone provide me with the code for it. I want to use the same dict in C++.

quantotto
@quantotto
@nikhilTkur docs: https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html#binding-stl-containers
The key is to use PYBIND11_MAKE_OPAQUE macro
Example showing how to make vector opaque: https://github.com/pybind/pybind11/blob/master/tests/test_opaque_types.cpp
For map the idea is the same:
PYBIND11_MAKE_OPAQUE(std::map<int, std::map<int, int>>);
PYBIND11_MAKE_OPAQUE(std::map<int, std::vector<std::tuple<>>>); // need to specify types for tuple
// later in binding code:
py::bind_map<std::map<int, std::map<int, int>>>(m, "MapIntToMap");
nikhilTkur
@nikhilTkur

@quantotto I tried that, however, I am trying to pass a py::dict as an input. For simplicity, lets say that I pass a py::dict (int : int ) and want to use it as an unordered_map in a function. Here is the code for it:

std::unordered_map<int, int> modify_map(py::dict dictionary) {
         std::unordered_map<int, int> mapp = dictionary.cast<std::unordered_map<int , int>>();
         return mapp;
}
PYBIND11_MODULE(SYCL, m) {
    m.def("mod", &modify_map);
    py::bind_map<std::unordered_map<int, int>>(m, "IntMap");
}

it gives the error: Unable to cast Python instance to C++ type (compile in debug mode for details).

any suggestions how to do this?

Adam Thompson
@ajum
I think I have a chicken and egg problem.... I'm defining submodules in my module, I'd like to be able to list the members of one submodule in a method in a member of another submodule, is that possible?
1 reply
quantotto
@quantotto

@nikhilTkur did you include <pybind11/stl.h>? When included, the code worked for me:

>>> import SYCL
>>> dct = {4: 1, 8: 2}
>>> SYCL.mod(dct)
{8: 2, 4: 1}
>>>

bindings code:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>

namespace py = pybind11;

std::unordered_map<int, int> modify_map(py::dict dictionary) {
         std::unordered_map<int, int> mapp = dictionary.cast<std::unordered_map<int , int>>();
         return mapp;
}
PYBIND11_MODULE(SYCL, m) {
    m.def("mod", &modify_map);
    py::bind_map<std::unordered_map<int, int>>(m, "IntMap");
}
Nam Vu
@Namburger
hi all, I'm Nam, new here, would like to ask what version of python3 is supported with pybind11/2.6.2?
quantotto
@quantotto
@Namburger it is in the docs: Python 2.7, 3.5+, and PyPy/PyPy3 7.3 are supported with an implementation-agnostic interface.
Kerim
@kerim371

Hi,
When binding templated function:

template<typename D, typename T>
Eigen::MatrixX<typename D::Scalar> TvdssXY2ALL(
    const Eigen::DenseBase<D> &M,
    const T& x0, const T& y0, const T& kb,
    const bool& XNorth);

I get compilation error:

C:\Users\tasik\Documents\Git_Projects\h5geo\build-MSVC2019-Release\_deps\pybind11-src\include\pybind11\cast.h:2030: error: C2280: 'bool pybind11::detail::type_caster<Eigen::DenseBase<Derived>,void>::load(pybind11::handle,bool)': attempting to reference a deleted function [C:\Users\tasik\Documents\Git_Projects\h5geo\build-MSVC2019-Release\src\h5geopy\_h5geo.vcxproj]
  with
  [
      Derived=Eigen::Matrix<double,-1,-1,0,-1,-1>
  ]

Maybe somebody could had something similar?

I do bindings with code:
m.def("TvdssXY2ALL", &h5geo::TvdssXY2ALL<Eigen::MatrixX<double>, double>);
Kerim
@kerim371
@quantotto Eigen::Matrix<>is a template and Eigen::MatrixXis a typedef for Eigen::Matrix<Eigen::Dynamic, ...>or something
8 replies
Pablo Hernandez-Cerdan
@phcerdan
My googling is failing, where do I put the documentation on a pybind11::class_?
Never mind, in the constructor, as usual. Sorry for the noise.
Ben
@benknight135
I have third party dll's that my library depend on, to make sure they are picked up by the python module I have been adding the directory of the dlls to the path. Is there a better way to do this?
4 replies
paulgessinger
@paulgessinger:matrix.org
[m]
hello! i have a C++ class that i expose to python, which runs a workload in multiple threads spawned from C++. ideally i want to run python functions from within those C++ threads. if my understanding of the GIL is correct, that's not possible. am i missing something?
6 replies
Taro
@rcffc

How do I bind a c++ std::pair to py::tuple? I have function that returns std::vector<std::pair<Vector3i, ITMVoxel>> which I bind with

.def("get_occupied_voxel_blocks", [](ITMLocalVBA<ITMVoxel> &vba, int size, ITMLib::ITMVoxelBlockHash::IndexData &index)
          { return py::array_t<py::tuple<Vector3i, ITMVoxel>>(
                {vba.occupied.size()},
                {sizeof(ITMVoxel)},
                vba.GetOccupiedVoxelBlocks(size, &index)); 
          })

However, pybind does not seem to like the way I bind the tuple. I can confirm that the rest works, it works without using pair.

[build] C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\pynfinitam.cpp(200,36): error C2947: expecting '>' to terminate template-argument-list, found '<' [C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\build\pynfinitam.vcxproj]
[build] C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\pynfinitam.cpp(210,42): error C2146: syntax error: missing ';' before identifier 'Vector3i' [C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\build\pynfinitam.vcxproj]
[build] C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\pynfinitam.cpp(210,24): error C2275: 'pybind11::array_t<pybind11::tuple,16>': illegal use of this type as an expression [C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\build\pynfinitam.vcxproj]

Google has not shown an example in which py::pair needs to be parametrized. But if I only write py::array_t<py::tuple>(...) then I will get the error

[build] C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\pynfinitam.cpp(210,42): error C2440: '<function-style-cast>': cannot convert from 'initializer list' to 'pybind11::array_t<pybind11::tuple,16>' [C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\build\pynfinitam.vcxproj]
[build] C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\pynfinitam.cpp(200,36): message : No constructor could take the source type, or constructor overload resolution was ambiguous [C:\Users\pejiang\Documents\GitHub\smg-pynfinitam\build\pynfinitam.vcxproj]
quantotto
@quantotto
@rcffc py::tuple is not a template class, hence you can't supply template args in the angle brackets. As for using tuple with array_t, I believe it is not supported since array_t expects only well known structure / size types. Data is stored in a continuous buffer and tuple doesn't fit into this paradigm. Probably, py::list should be used instead.
Taro
@rcffc
@quantotto Thanks for your answer. Alright, I'm gonna try py::list
Taro
@rcffc
@quantotto Using py::list lead to the same last error (C2440). Also when using py::array_t with a struct that includes the properties Vector3i and ITMVoxel. :/
quantotto
@quantotto
@rcffc could you share the new code? It is not a simple substitution (py::list in place of py::array_t). Maybe I didn't explain it correctly. I meant that you need to switch to python list object. It may require returning std::vector<py::tuple> on C++ side and for Python it will be presented as list object.
Taro
@rcffc
@quantotto Yeah I just substituted it as you said
I am gonna try that

return py::array_t<ITMLocalVBA<ITMVoxel>::VectorVoxelPair>( {vba.occupied.size()}, {sizeof(ITMVoxel)}, vba.GetOccupiedVoxelBlocks(size, &index));

struct VectorVoxelPair
        {
            Vector3i vector;
            TVoxel voxel;
            VectorVoxelPair(Vector3i ve, TVoxel vo)
            {
                vector = ve;
                voxel = vo;
            }
        };

This is the other way I tried

quantotto
@quantotto
@rcffc going with the struct approach requires also registering DTYPE (there is special macro for that) and, besides, each type used in the struct should be registered too. It is probably not possible for your case, hence using vector / list is more appropriate (unless there is a way to re-arrange your data to fit into an array of more basic types supported by numpy).
Taro
@rcffc
Yep, I did register the struct with NUMPY_DTYPE
@quantotto And the subtypes were already registered (ITMVoxel, Vector3i)
I am trying to just split return two arrays and access them separately as a workaround.
quantotto
@quantotto
@rcffc I think they support only record type structs. If it is an interface or has methods, then it is not going to work
Taro
@rcffc
I see. But doesn't the one I made count as record type struct? Unless the constructor counts as method as well...
quantotto
@quantotto
Depends how Vector3i and TVoxel are defined.
Taro
@rcffc
Ah, so those also may not have methods? Vector3i has a lot of methods, well...
quantotto
@quantotto
If the type can't go into numpy array, then it won't work. In the examples, all types are pretty basic and allow nesting of simple structs. https://github.com/pybind/pybind11/blob/e08a58111dbea38d667b209f7543864d51a3b185/tests/test_numpy_dtypes.cpp
Angus Gibson
@angus-g
Is it possible to call pickling functions from c++? e.g. I have a C++ class A that holds a shared_ptr to a trampoline class with Python overrides PyB, and I want to serialise my A: then I need to load the right Python type back (that is, PyB on its own doesn't have the required Python state to de-serialise symmetrically)
41 replies
Ghost
@ghost~60c85d746da03739847ee8ae
Hello! I'm very much new to pybind11, and getting following error with PYBIND11_MODULE when trying to create a simple sample project. What is it that I am missing?
5 replies
errormsg.png
Ujjwal Panda
@astrogewgaw
hey folks! i am sure this question has been asked before, but i need to write code that works across multiple Numpy dtypes. now i know that pybind11's Numpy bindings can implicitly convert types, but i want to use Numpy's memmap to process data that is too large to read into my RAM at one time, and for a memory-mapped file any and all type conversions fail with a Memory Error. is it possible to use a template, or some other method, to make my code accept a Numpy array of any dtype without type conversion? i know that py::array_t exists, but how do i use it for something like this? if anyone wants to check out what type of code i am working with here, it is something like this: https://github.com/astrogewgaw/pkern/blob/main/src/pkern/cpp/dedisp.hpp; checkout the dedisp function.
crimsoncor
@crimsoncor

I can do tests myself, but was wondering if anyone knew if the default stl_bind methods like bind_vector handle things like vectors of vectors cleanly from the python side.
so if I've got a c++ field that is std::vector<std::vector<std::string>> listOfLists can I do something like

s.getListOfLists().append(["cat", "dog"])

or am I forced to use the Python wrapper type instead

lvk88
@lvk88

Hi all! I've got a question to you about best practices for memory errors and leak detection. I am working on a (mostly) C++ project for numerical computations with bindings to python through pybind11. There are a lot of unit tests involving numpy. I am trying to enable leak detection now, using address sanitizer from GCC. I have a custom-built python interpreter with built with the --with-address-sanitizer and --with-undefined-behavior-sanitizer flags. Those tests that do not involve numpy are passing without issues, but I am still having problems to understand the error messages that appear when importing numpy. Even if I start my custom python and do a simple import numpy, and exit right away, I am getting a lot of Indirect leak of ... byte(s) in 1 object(s) allocated from ... messages. Most of them involve the random component of numpy.

Are there best practices to be followed when running memory error detection tools on python code? Is there a repo with common false positives coming from numpy? I had a look on the valgrind suppression files in the test folder of the pybind11 repo, and there is not a 100% overlap between the suppressions and error messages I am getting. Would valgrind be a better choice for memory error detection? In the first round I disregarded valgrind because my past experience shows that it adds quite some runtime performance overhead in comparison to asan.

Jerry Johns
@mrjerryjohns
Hello! Is there a way to dump out the fields and values of a bound C++ object that has had its members bound using def_readwrite? Tried accessing .dict and vars() to no avail…
1 reply
ripare
@ripare
Hello guys, I have a situation where the ref count of a cpp instantiated array is not being decremented. More info in the thread. :arrow_down_small:
7 replies