Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    echorand (Amit Saha)
    @amitsaha
    yeah that is my first plan
    echorand (Amit Saha)
    @amitsaha
    Q - how do you edit Pyodide code? So far, i have tried vim and vscode and the indentation and syntax highlighting leaves much to be desired?
    2 replies
    Roman Yurchak
    @rthy:matrix.org
    [m]

    Nice post!

    Just a comment that this would only work when downloading textual files. Downloading binary files can only be done asynchronously in the browser (e.g. with fetch) and in that case monkeypatching a sync function with it wouldn't work.

    echorand (Amit Saha)
    @amitsaha
    Thanks Roman, will think of an update with that!
    Roman Yurchak
    @rthy:matrix.org
    [m]
    Re: editor, vim for me, and vscode for Hood (I'm not sure what other devs use). But yes in any case there are likely suboptimal behavior due to a unexpected mix of languages, sometimes in the same file.
    echorand (Amit Saha)
    @amitsaha
    Gotcha
    I am also confused about how exactly to align my Python code, do i have to make sure that the Python code is always left aligned and then the usual indentation applies? However I think i have seen that the left alignment is not necessary as long as the rest of the code is indented correctly
    Roman Yurchak
    @rthy:matrix.org
    [m]
    We run textwrap.dedent on the code, so the indentation of the first line is removed, and the rest should have correct indentation with respect to the first line. This also means you can shift the indentation by the same amount, without it impacting execution
    echorand (Amit Saha)
    @amitsaha
    ah great thanks that's what i wanted to know!
    Thorsten Beier
    @DerThorsten
    some fun stuff of mine: pyb2d as pyodide package + jupyterlite = https://twitter.com/ThorstenBeier/status/1523573928702087169
    Rick Katka
    @rkatka
    Am I able to pass multiple package names when using pyodide_build mkpkg and pyodide_build buildpkg?
    Michael Droettboom
    @mdboom
    There is a new "Category" for WebAssembly in the main Python discourse forum. This is a place to coordinate between the various Python+WebAssembly-related projects and make sure we are all pointing in the same direction when possible. (So not intended to replace this pyodide-specific forum, but to be a place to make sure pyodide continues to work will with pyscript and jupyterlite and other dependencies etc.) I encourage anyone interested to join: https://discuss.python.org/c/webassembly
    stonebig
    @stonebig
    Is there a timeline or condition for next pyodide release ? Will pyodide get the same speed boost with python-3.11b1 as cpython ?
    Roman Yurchak
    @rthy:matrix.org
    [m]

    Pyodide has generally a stable release every 3 moths. Last one was 1 month ago, so the next one would likely happen in July. That's for stable release, we are trying to also do more frequent alpha releases in between.

    For Python 3.11 we are expecting to update to it once there is a stable release in October 2022 and packages officially support it (a few month later). So it will happen by the end of the year. Yes, we will likely see the speed boost, but there is also likely room for WASM specific optimization as currently the difference in performance with native performance is still quite large (3x-5x) while it is much less for C code in general.

    stonebig
    @stonebig
    Ok,but what if you discover only in october a bottleneck in cpython 3.11 ? Wouldn't it make sense to generate an alpha over the beta to see how it fly ? If Sheldon team can tweak cpython for you a bit ?
    stonebig
    @stonebig
    Maybe it's more an exploration for anaconda team, but they tend also to wait after rc
    Hood Chatham
    @hoodmane
    Well I have built Pyodide against Python 3.11 a6. We don't have benchmarks because our benchmark suite mostly relies on numpy which hasn't been updated to build against Python 3.11 yet.

    Will pyodide get the same speed boost with python-3.11b1 as cpython ?

    Most likely we will get a similar speed boost, the last few updates we have gotten similar speed benefits to native Python (we're consistently 2x to 3x slower)

    Wouldn't it make sense to generate an alpha over the beta to see how it fly ?

    It would make sense, but we are very limited in development resources so it may or may not happen

    Ryan May
    @dopplershift
    I'm trying to get PROJ building, which requires sqlite3 >=3.11 I know Pyodide includes it with Python, but anyone have any clue if that's something that can be linked to?
    Roman Yurchak
    @rthy:matrix.org
    [m]
    @dopplershift: if you are talking about another package that needs SQLite at link time. I imagine its object files should still be in the CPython build directory. So it should be possible to statically link it to your application. Otherwise a better solution would be it make a dynamic library that would be unvendored from the stdlib as we are doing with distutils currently
    Ryan May
    @dopplershift
    @rthy:matrix.org Yeah, it's a compiled library that needs it at link time (and the headers for build). I'll see what I can do to point it at those files in the build dir.
    Hood Chatham
    @hoodmane
    We statically link sqlite3 so you probably shouldn't link it again because it increases download sizes and may lead to symbol name clashes. I think you should add -I $PYODIDE_ROOT/cpython/build/sqlite-autoconf-3380500/ to cflags and that might be all you need.
    prescod
    @prescod_twitter
    So I'm trying to understand the workflow that one usually uses if I have a pre-existing package with a lot of requirements. My package's requirements are slight variants on the ones that Pyodide comes with. So one technique I tried was to compile each of their preferred versions into a whl and put them in the "dist" directory. But micropip doesn't seem to detect that has already installed a compatible wheel
            import micropip
            await micropip.install('./python_baseconv-1.2.2-py3-none-any.whl')
            import baseconv
            print("SUCCESS INSTALLING BASECONV")
            await micropip.install(['./python_baseconv-1.2.2-py3-none-any.whl', './snowfakery-3.1.0-py3-none-any.whl'], keep_going=True)
          `);
    I get to the success message but the next line complains:
      File "/lib/python3.10/site-packages/micropip/_micropip.py", line 189, in install
        raise ValueError(
    ValueError: Couldn't find a pure Python 3 wheel for: 'greenlet==1.1.2', 'markupsafe==2.0.1', 'sqlalchemy==1.4.36', 'python-baseconv==1.2.2'
    The version of python-baseconv it complains about is the one it just successfully loaded (twice!).
    Hood Chatham
    @hoodmane
    This looks like a bug @prescod_twitter. I think it is getting confused between python_baseconv and python-baseconv.
    Hood Chatham
    @hoodmane
    We ought to be using packaging.utils.canonicalize_name
    Hmm we use packaging.normalize_package_name is that different?
    Oh it's our own reimplementation...
    Hood Chatham
    @hoodmane
    The specific bug you are hitting is that this line:
    https://github.com/pyodide/pyodide/blob/main/packages/micropip/src/micropip/_micropip.py#L305
    if req.name in transaction["locked"]: should be if normalize_package_name(req.name) in transaction["locked"]:
    Hood Chatham
    @hoodmane
    I think this should fix it:
    pyodide/pyodide#2560
    prescod
    @prescod_twitter
    Thanks for the quick fix!
    In general do you have a good technique or tool for recursively bundling the right versions of packages to support a top-level package?
    Hood Chatham
    @hoodmane

    No, currently I think it is annoying.

    From the design perspective, my plan is to add a micropip.freeze api that creates a packages.json lock file with information about the currently installed packages. Then loadPyodide would be updated to accept a url to the custom lock file and you could install your package and deps with pyodide.loadPackage. Ideally we would take also a list of packages to load as an argument to loadPyodide so we can load those packages while we are loading / compiling the wasm.

    In the meantime, I'm not sure what the best approach is.

    echorand (Amit Saha)
    @amitsaha
    Hi @hoodmane @rthy:matrix.org -- looking to pick up a new issue for me to chip at as and when I find the time, do you have any recommendations?
    Roman Yurchak
    @rthy:matrix.org
    [m]
    Hi @amitsaha I think micropip related features could be a possibility for instance pyodide/pyodide#2535 (or other related issues) There is a lot of user demand for it and you are already familiar a bit with that part of the code base. Of course if there are other areas you would rather work on we can certainly also find issues there.
    echorand (Amit Saha)
    @amitsaha
    Sounds great, i will take it up!
    Drew Siedel
    @dtsiedel

    Hi all, I have a question, for which I hope this is the right forum. Please let me know if an issue would be more appropriate. I'm trying to grab a package (h5py) in pyscript, and I see that it's been added to the packages list on pyodide's GitHub, but is not in the released version v0.20.0 that pyscript references.

    I was hoping to just grab the wheel and statically host it for my pyscript site, while waiting for it to be added to the list. How would I go about downloading a copy of the python-only wheel for h5py? I see https://github.com/pyodide/pyodide/tree/main/pyodide-build, which looks like it might have the tools to build it from the configs listed at https://github.com/pyodide/pyodide/tree/main/packages?

    Hood Chatham
    @hoodmane
    Well you can get the wheel here:
    https://cdn.jsdelivr.net/pyodide/dev/full/h5py-3.6.0-cp310-cp310-emscripten_wasm32.whl
    I think
    await pyodide.loadPackage(["numpy", "https://cdn.jsdelivr.net/pyodide/dev/full/h5py-3.6.0-cp310-cp310-emscripten_wasm32.whl"])
    seems to work in v0.20.0, but there are no ABI compatibility checks so if we update the compiler it would stop working. But if you grab that wheel and host it statically from your own side then it would probably work.
    Drew Siedel
    @dtsiedel
    Thanks for the info @hoodmane! It seems like pyscript is expecting py3-none-any wheels though, since it gives me ValueError: 'h5py-3.6.0-cp310-cp310-emscripten_wasm32.whl' is not a pure Python 3 wheel when I dropped that wheel into my static dir and added it in the py-env section. Does pyodide only build wasm wheels, like the one you linked from the cdn? Since pyscript is depending on pyodide for these wheels, it feels like they must work on the same wheel type, but maybe it's more complicated than I was thinking
    Hood Chatham
    @hoodmane
    try:
    <py-script>
    from pyodide_js import loadPackage
    await loadPackage(["numpy", "https://cdn.jsdelivr.net/pyodide/dev/full/h5py-3.6.0-cp310-cp310-emscripten_wasm32.whl"])
    </py-script>
    The wheel type depends on the package, if the package is Python-only, then we get py3-none-any, but if it has C, C++, Rust, cython, etc etc then it can only produce binary emscripten_wasm32 wheels (except for a small number of packages that have pure Python slow implementations and implement the same functionality in C, these can have both binary and pure python wheels).
    The error you get is because the py-env section gets loaded with micropip which currently cannot handle binary wheels. This limitation kind of makes sense because micropip is primarily for downloading wheels from PyPI which does not accept emscripten-wasm wheels, but it is a bit annoying. Hopefully in the future we will fix it to accept binary wheels too.
    Drew Siedel
    @dtsiedel
    Great, that seems to work, thank you :smile:. And thanks for the info on the different wheel types, that makes sense. So if I have another package that depends on h5py, should I also dynamically load it that way? It seems like py-env wouldn't be able to see h5py since I grabbed it down in the script section
    Hood Chatham
    @hoodmane
    Right, I'm not sure if <py-script> is willing to accept multiple <py-env> sections. You could do:
    import micropip
    from pyodide_js import loadPackage
    await loadPackage(["numpy", "https://cdn.jsdelivr.net/pyodide/dev/full/h5py-3.6.0-cp310-cp310-emscripten_wasm32.whl"])
    await micropip.install("some-other-package")
    Drew Siedel
    @dtsiedel
    Great, thanks again :grinning:
    prescod
    @prescod_twitter
    Thank you for Pyodide. It's amazing!
    Henry Schreiner
    @henryiii
    This is moving forward again, looks very promising! executablebooks/thebe-core#3