2 replies
Nice post!
Just a comment that this would only work when downloading textual files. Downloading binary files can only be done asynchronously in the browser (e.g. with fetch) and in that case monkeypatching a sync function with it wouldn't work.
Pyodide has generally a stable release every 3 moths. Last one was 1 month ago, so the next one would likely happen in July. That's for stable release, we are trying to also do more frequent alpha releases in between.
For Python 3.11 we are expecting to update to it once there is a stable release in October 2022 and packages officially support it (a few month later). So it will happen by the end of the year. Yes, we will likely see the speed boost, but there is also likely room for WASM specific optimization as currently the difference in performance with native performance is still quite large (3x-5x) while it is much less for C code in general.
Will pyodide get the same speed boost with python-3.11b1 as cpython ?
Most likely we will get a similar speed boost, the last few updates we have gotten similar speed benefits to native Python (we're consistently 2x to 3x slower)
Wouldn't it make sense to generate an alpha over the beta to see how it fly ?
It would make sense, but we are very limited in development resources so it may or may not happen
import micropip
await micropip.install('./python_baseconv-1.2.2-py3-none-any.whl')
import baseconv
print("SUCCESS INSTALLING BASECONV")
await micropip.install(['./python_baseconv-1.2.2-py3-none-any.whl', './snowfakery-3.1.0-py3-none-any.whl'], keep_going=True)
`); File "/lib/python3.10/site-packages/micropip/_micropip.py", line 189, in install
raise ValueError(
ValueError: Couldn't find a pure Python 3 wheel for: 'greenlet==1.1.2', 'markupsafe==2.0.1', 'sqlalchemy==1.4.36', 'python-baseconv==1.2.2'python-baseconv it complains about is the one it just successfully loaded (twice!).
packaging.normalize_package_name is that different?
https://github.com/pyodide/pyodide/blob/main/packages/micropip/src/micropip/_micropip.py#L305
if req.name in transaction["locked"]: should be if normalize_package_name(req.name) in transaction["locked"]:
No, currently I think it is annoying.
From the design perspective, my plan is to add a micropip.freeze api that creates a packages.json lock file with information about the currently installed packages. Then loadPyodide would be updated to accept a url to the custom lock file and you could install your package and deps with pyodide.loadPackage. Ideally we would take also a list of packages to load as an argument to loadPyodide so we can load those packages while we are loading / compiling the wasm.
In the meantime, I'm not sure what the best approach is.
Hi all, I have a question, for which I hope this is the right forum. Please let me know if an issue would be more appropriate. I'm trying to grab a package (h5py) in pyscript, and I see that it's been added to the packages list on pyodide's GitHub, but is not in the released version v0.20.0 that pyscript references.
I was hoping to just grab the wheel and statically host it for my pyscript site, while waiting for it to be added to the list. How would I go about downloading a copy of the python-only wheel for h5py? I see https://github.com/pyodide/pyodide/tree/main/pyodide-build, which looks like it might have the tools to build it from the configs listed at https://github.com/pyodide/pyodide/tree/main/packages?
https://cdn.jsdelivr.net/pyodide/dev/full/h5py-3.6.0-cp310-cp310-emscripten_wasm32.whl
I think
await pyodide.loadPackage(["numpy", "https://cdn.jsdelivr.net/pyodide/dev/full/h5py-3.6.0-cp310-cp310-emscripten_wasm32.whl"])seems to work in v0.20.0, but there are no ABI compatibility checks so if we update the compiler it would stop working. But if you grab that wheel and host it statically from your own side then it would probably work.
ValueError: 'h5py-3.6.0-cp310-cp310-emscripten_wasm32.whl' is not a pure Python 3 wheel when I dropped that wheel into my static dir and added it in the py-env section. Does pyodide only build wasm wheels, like the one you linked from the cdn? Since pyscript is depending on pyodide for these wheels, it feels like they must work on the same wheel type, but maybe it's more complicated than I was thinking
py3-none-any, but if it has C, C++, Rust, cython, etc etc then it can only produce binary emscripten_wasm32 wheels (except for a small number of packages that have pure Python slow implementations and implement the same functionality in C, these can have both binary and pure python wheels).
py-env section gets loaded with micropip which currently cannot handle binary wheels. This limitation kind of makes sense because micropip is primarily for downloading wheels from PyPI which does not accept emscripten-wasm wheels, but it is a bit annoying. Hopefully in the future we will fix it to accept binary wheels too.
py-env wouldn't be able to see h5py since I grabbed it down in the script section
<py-script> is willing to accept multiple <py-env> sections. You could do:import micropip
from pyodide_js import loadPackage
await loadPackage(["numpy", "https://cdn.jsdelivr.net/pyodide/dev/full/h5py-3.6.0-cp310-cp310-emscripten_wasm32.whl"])
await micropip.install("some-other-package")