PyO3/Lobby

Help for and discussion about the Python/Rust interface library PyO3.

Bonts

@bonts:matrix.org

katetsu (Hoss Ajallooiean) do you use maturin ?

Matt Smith

@matt24smith

the maturin docs state "To include arbitrary files in the sdist for use during compilation specify sdist-include [in pyproject.toml] as an array of globs": how do i find these paths after compilation? sorry if this is a noob question

[tool.maturin]
sdist-include = ["path/**/*"]

Matt Smith

@matt24smith

figured it out, had to move files into python source dir, not package root

Lucas Yunkyu Lee

@RocketLL

What’s the preferred way to have a Rust extension that also can function as a standalone Rust binary/library?

Matt Smith

@matt24smith

@RocketLL I think what you want is something like the following in Cargo.toml

[lib]
crate-type ["rlib", "cdylib"]
path = "src/lib.rs"

[[bin]]
path = "src/bin.rs"
...

mejrs

@mejrs

@RocketLL you can have it all in a single crate, but I would recommend creating a workspace and just have an extension module crate that wraps the main crate

the best solution depends on how boiler plate-y stuff gets and how much if anything the python api needs to reach into the main lib's internals

also, mixing pyclasses between crates doesn't work well and should be avoided

Lucas Yunkyu Lee

@RocketLL

@mejrs @matt24smith Thanks!

mikeWShef

@mikeWShef

Hey does anyone know the right way to pass an object with a set trait to a pymethod? I have somthing like this code:

trait LinearSystem{
    fn mat_vec(&mut self, input:&ArrayView1<f64>, output:&ArrayView1<f64>)
}

type LinearSystemType = Box<dyn LinearSystem>

#[pyclass]
struct System1 {
    ...
}

impl LinearSystem for System1 {
    fn mat_vec(...){
        ...
    }
}

#[pyclass]
struct Solver{
    ...
}

#[pymethods]
impl Solver{
    solve(&mut self, system: LinearSystemType)
}

But because Solver.solve is in pymethods Linear System also needs to be PyClass, if I try to combine them I have to fill in all the associated types from py03's internals which feels like the wrong way to go, is there a common way to do this?

mejrs

@mejrs

you can probably make a pyclass struct and put the boxed trait object inside it

when you write a python method, pyo3 has to know how to create rust objects from python objects, and it can't create boxed trait objects from them

mikeWShef

@mikeWShef

Ah thanks, that has made the whole python interface much simpler too!

kai

@kblin:matrix.org

hi folks. This might be a silly question but I can't find it in the docs. I've got a function like pub fn get_name_by_id(&self, id: &PyLong, allow_deprecated: &PyBool) -> PyResult<String> {...}, and I'd like to set the default value of allow_deprecated to False

I read in the docs that in #[args(allow_deprecated = "false")] the value should be a valid Rust type

but now I'm getting an error about bool not being convertible to a &PyBool

so what do I pass in?

mejrs

@mejrs

what about just taking bool as the argument?

kai

@kblin:matrix.org

uh, too easy? ;) Kidding aside I didn't realise this was an option

that works

kai

@kblin:matrix.org

or rather: this compiles, but I'm getting a segfault trying to use it

kblin

@kblin:matrix.org

ah, unrelated bug in my error handling code, go figure

Ivan Canet

@CLOVIS-AI

Hi! Is there an easy way to generate some kind of documentation report (e.g. an HTML page) that includes Rust functions exported to Python?

I'm finding a few things for C extensions, but I doubt they'll work on Rust code

konstin

@konstin:matrix.org

is there a way to load a specific python version, inject some python code and ideally some rust callbacks and then run a module/script as you would by calling pythonx.y directly?

mejrs

@mejrs

I'm finding a few things for C extensions, but I doubt they'll work on Rust code

It's worth a try, if this works on the compiled libraries

most tools like valgrind etc tend to work out of the box on Rust

Kevin Heavey

@kevinheavey

I'm curious why nobody else seems to mind that Maturin doesn't support a directory of .pyi files (instead of just a single stub file) PyO3/maturin#792

I think it severely limits what I can do but since I've never seen anyone else mention it I'm wondering if I'm missing something that other people are doing

konstin

@konstin:matrix.org

i think I've simply never seen anyone use more than more then one pyi file before

1 reply

konstin

@konstin:matrix.org

yes, or you re-export everything public in the root module or python part

messense

@messense

I'm curious why nobody else seems to mind that Maturin doesn't support a directory of .pyi files (instead of just a single stub file) PyO3/maturin#792

Is stub-only packages suitable for you? https://peps.python.org/pep-0561/#stub-only-packages

Reggie

@reggiemarr:bridge.monster

Hi all, I have a rust library I'd like to expose as a python library which I had previously done using CFFI. This seems to be a better option however I can't seem to figure out how to get maturin to use cross instead of cargo. Is there some reason why this is non-trivial ?

David Hewitt

@davidhewitt

why do you need cross? Maturin actually has pretty good cross compiling support with PyO3's abi3 feature natively

Reggie

@reggiemarr:bridge.monster

Cross would be useful since it allows me easily utilize a docker container which packages up the dependencies I'm using

Ivan Canet

@CLOVIS-AI

What's the recommendation for #[pyclass]?
Should it be put on the domain object, or on an Arc<Mutex<ThatDomainObject>>

Reggie

@reggiemarr:bridge.monster

yeah without using cross or some clearer documentation around docker usuage I'm not really sure how anyone is cross compiling with this tool

1 reply

Seemingly its being done but the process to pack in dependencies is pretty opaque

Reggie

@reggiemarr:bridge.monster

I know how to make this work with cross but none of the associated docker images with this project are working for me and I can't seem to build off them

Reggie

@reggiemarr:bridge.monster

Isnt that just for use with a gitrepo ? What if we want to build locally

1 reply

GitHub repo*

Reggie

@reggiemarr:bridge.monster

Hmm that's bizarre because unless your compiling with no depenencies this just doesn't seem to work

1 reply

Reggie

@reggiemarr:bridge.monster

So for example my project uses serialport which utlizes libudev.

using cross I would normally just install the appropriate libudev for arm 64 but the following fails when trying to use the manylinux (or pyo3) container

FROM messense/manylinux_2_24-cross:aarch64


RUN curl https://sh.rustup.rs -sSf | bash -s -- -y

ENV PATH="/root/.cargo/bin:${PATH}"

# Fails here
RUN rustup target add aarch64-unknown-linux-gnu
RUN apt-get update
RUN dpkg --add-architecture arm64 && apt-get update

RUN apt-get install --assume-yes libudev-dev:arm64 && \
    apt-get install --assume-yes pkg-config-aarch64-linux-gnu

ENV PKG_CONFIG_LIBDIR_aarch64_unknown_linux_gnu=/usr/aarch64-linux-gnu/lib/pkgconfig

Reggie

@reggiemarr:bridge.monster

But when I try to build I get the following:

  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "`\"pkg-config\" \"--libs\" \"--cflags\" \"libudev\"` did not exit successfully: exit status: 1\nerror: could not find system library 'libudev' required by the 'libudev-sys' crate\n\n--- stderr\nPackage libudev was not found in the pkg-config search path.\nPerhaps you should add the directory containing `libudev.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'libudev' found\n"', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/libudev-sys-0.1.4/build.rs:38:41

Reggie

@reggiemarr:bridge.monster

I should note that the docker container fails to build with the add arch line

Reggie

@reggiemarr:bridge.monster

Trying to build with the pyo3 like so ❯ docker run --rm -v $(pwd):/io ghcr.io/pyo3/maturin build --release --target aarch64-unknown-linux-gnu

Results in something like the following:

error: aborting due to 1039 previous errors


Some errors have detailed explanations: E0405, E0408, E0412, E0422, E0425, E0432, E0433, E0463, E0531...

For more information about an error, try `rustc --explain E0405`.

error: build failed
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `cargo rustc --manifest-path Cargo.toml --message-format json --release --lib --`

streamio on  master [$»!?] is 📦 v0.1.0 via 🐍 v3.8.12 via 🦀 v1.63.0-nightly via C base took 1m45s 
❯ docker run --rm -v $(pwd):/io ghcr.io/pyo3/maturin build --release --target aarch64-unknown-linux-gnu

Also I'm not sure if I should actually be making an issue about this but I just figured I was doing something obviously wrong