ˈt͡sɛːzaɐ̯ If I try to glob() to look around a bit, I just get emscription assertions…
1 reply
ˈt͡sɛːzaɐ̯ hSo the writing files part of downloading skyfield data is not a problem at all…
ˈt͡sɛːzaɐ̯ I just wondered: does micropip work in web workers?
ˈt͡sɛːzaɐ̯ Ah, it's easy to fix… http://dpaste.com/2QD5P61 Guess I'll submit a PR for that later. At least now I know how to do the data downloads for my skyfield PR. :)
For the people who have ongoing projects using pyodide, how likely is it that the project will become unmaintained?
ˈt͡sɛːzaɐ̯ Hm, I've tried to make a pyodide package for something with a C library, like so: https://doggieworld.liftm.de/test/pyodide/mecab/mecab.txz . Problem is: it seems to throw a number as an exception, like here https://doggieworld.liftm.de/test/pyodide/mecab/wwmecab.html and as soon as it enters the C++ parts. How do I debug that kind of thing?
ˈt͡sɛːzaɐ̯ part was my fault, because I tried to build that library with skip_host and emconfigure, which bypasses the symlinks of pywasmcross. but it also fails because the build command capture opens build.log in the current working directory, and that changes during compiling...
Hi, my name is Kenji, an engineer at brilliant.org.
I started using pyodide a bit, and it's looking pretty exciting.
I realized that scipy is pretty heavy and I wanted to see if I can do something.
The issue is explained here by rth, with a solution proposal from navytux:
So far, the things I've played around are:
(a) split scipy.data into smaller chunks
It looks like Google Chrome does not cache files over a certain size, and scipy.data
is above the threshold. To force the cache, I tried separating scipy.data into smaller
chunks and tweak the fetchRemotePackage() function in scipy.js file to concatinate them.
This helps the cache situation, but the first access will still remain to be heavy.
(b) remove some scipy subpackages
Delete a few config.add_subpackage() lines in scipy/setup.py and build a custom scipy
to see how much it helps reduce the size of scipy.data. This reduces the size but scipy
still remains to be pretty large. I think this is due to LAPACK and BLAS.
(c) play around with DSO
I tried playing around with DSO -> DSO linking in emscripten. I'm not familiar enough
but at least started to play with it.
So, the next steps I want to try is to dynamically link LAPACK and BLAS. I am assuming
LAPACK is at pyodide/CLAPACK/CLAPACK-WA/lapack_WA.bc as an archive and that gets
statically linked and placed in scipy.data. Building scipy isn't done through a
typical Makefile so it's a bit hard for me to track down how things are built, as I'm
not familiar with it.
May I ask for pointers or ideas of where I can look at to make dynamically linking work?
ˈt͡sɛːzaɐ̯ The magic all happens in pywasmcross It
- Symlinks itself to be invoked instead of the normal compilers (cc, ld, ..)
- Invokes the packages normal
setup.pyand captures all invocations to cc/ld/.. in a log - Replays the invocations, but this time with emcc/emld instead.
- Invokes
setup.py install --skip-build. The output of that is packaged into the .data file by buildpkg.py
Note that scipy has various patches that get applied before build: https://github.com/iodide-project/pyodide/tree/master/packages/scipy/patches
ˈt͡sɛːzaɐ̯ Ugh. Reading through all of iodide-project/pyodide#240 and a bit of emscripten-core/emscripten#7512 it seems like this will be a lot of fun.. you'll have to change all those static linking hacks into -s RUNTIME_LINKED_LIBS hacks.. いやだいやだ。
LoL. you understand Japanese!
Thanks for the explanation. I looked at the build.log file and got an understanding of what you meant. I was looking at the pyodide/packages/scipy/build/scipy-0.17.1/scipy/linalg/setup.py file, and noticed that deleting a config.add_extension('_flapack', ...) call will remove one static link. Of course that means things LAPACK won't work, but that gave me a hint on how things are built. (I'm basically trying to understand things by poking around)
May I ask what the static linking hacks you mentioned are? (or where I can find one or two)
ˈt͡sɛːzaɐ̯ This bit for example: https://github.com/iodide-project/pyodide/blob/96adc4b82d2dd4fc22cb8f8676ce6c57283fbfc8/pyodide_build/pywasmcross.py#L266
However, when I try to build a PyPI package that has dependencies, it fails to compile dependent packages. Is there a way to instruct pyodide to skip the compilation of dependecies?
@frthjf Could you please open an issue about it with the detailed log? As far I understand it would run setup.py install and at this point dependencies should already have been installed. If it's not the case, something might indeed be wrong with the dependency resolution.
@jurasofish Nice project! Maybe we could have a section on related projects somewhere in the documentation...
1 reply
@frthjf if you're looking to embed a pyodide component then my pyWeb might be similar to what you're after https://github.com/jurasofish/pyweb
Very cool, this looks like a good starting point, thank you!
jcaesar rth: Btw, is there anything missing for merging https://github.com/iodide-project/pyodide/pull/659#issuecomment-630549846 ?
