by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 16:50
    Heroku [iodide] deployed 217a7c17
  • Jun 30 23:38

    renovate[bot] on major-mui-monorepo

    Update mui monorepo (compare)

  • Jun 30 19:52

    renovate[bot] on react-table-7.x

    Update dependency react-table t… (compare)

  • Jun 30 16:37

    rth on master

    Compile python lxml (#544) Co-… (compare)

  • Jun 30 13:18

    renovate[bot] on babel-monorepo

    Update babel monorepo to v7.10.4 (compare)

  • Jun 30 09:25

    rth on master

    DOC corrected contribution link… (compare)

  • Jun 29 05:58

    dependabot-preview[bot] on pip

    (compare)

  • Jun 29 05:58

    dependabot-preview[bot] on pip

    Bump importlib-metadata from 1.… (compare)

  • Jun 28 22:21
    Heroku [iodide-staging] deployed 2b186c65
  • Jun 28 22:02

    wlach on master

    test_from_template_view (#2895)… (compare)

  • Jun 28 18:24

    phorward on master

    STY Apply black for code style … (compare)

  • Jun 28 13:51

    rth on master

    BLD Build CLAPACK, cpython and … (compare)

  • Jun 27 15:14

    rth on master

    Update to numpy 1.15.4 and pand… (compare)

  • Jun 27 12:57

    rth on master

    MNT Add pre-commit config (#698) (compare)

  • Jun 27 10:28

    rth on master

    Add package freesasa and add bi… (compare)

  • Jun 27 10:24

    rth on master

    MNT Don't benchmark scipy load … (compare)

  • Jun 27 06:37

    renovate[bot] on eslint-plugin-import-2.x

    Update dependency eslint-plugin… (compare)

  • Jun 26 15:33
    Heroku [iodide-staging] deployed 89e9a993
  • Jun 24 05:39

    dependabot-preview[bot] on pip

    (compare)

  • Jun 24 05:39

    dependabot-preview[bot] on pip

    Bump wcwidth from 0.1.9 to 0.2.… (compare)

William Lachance
@wlach
unfortunately we're a bit low on resources at the moment, so if you have any time/inclination to look into adding this feature (assuming it doesn't exist), that would be most welcome
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Btw, does pyodide by default have some writable folder?
ˈt͡sɛːzaɐ̯ If I try to glob() to look around a bit, I just get emscription assertions…
1 reply
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Hm, reading file_packager.py, /lib/python$ver/site-packages/$pkg should be writable..
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Thank you.
ˈt͡sɛːzaɐ̯ hSo the writing files part of downloading skyfield data is not a problem at all…
ˈt͡sɛːzaɐ̯ I just wondered: does micropip work in web workers?
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Doesn't seem like it: uncaught exception: <urlopen error unknown url type: https>
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ http://dpaste.com/247CZNG just for reference on what I'm trying…
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Ah, it's easy to fix… http://dpaste.com/2QD5P61 Guess I'll submit a PR for that later. At least now I know how to do the data downloads for my skyfield PR. :)
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Oh, nice, sys.setprofile works in pyodide. (Sadly, it didn't help. I think I need the equivalent of strace…)
Devin Neal
@eulertour
It seems like @mdboom, who I believed to be leading the pyodide project, is no longer working on it https://github.com/iodide-project/pyodide/issues/635#issuecomment-613408912.
For the people who have ongoing projects using pyodide, how likely is it that the project will become unmaintained?
William Lachance
@wlach
I guess it depends on what you mean by unmaintained, I've been reviewing things here and there along with @rth but substantial forward progress has slowed. Of course it's open source, anyone is free to fork the project and publish independently
Devin Neal
@eulertour
It's good that PRs are getting reviewed, and any forward progress at all is a plus.
Do any of the other maintainers plan to leave the project i.e. stop reviewing PRs?
Roman Yurchak
@rth
It's not very different from other OSS projects after some amount of time, there are ups and downs in maintainers availability unfortunately. I am trying to review pyodide PRs as well.
Paul m. p. P.
@pmp-p
hi all, @SimonBiggs i'd to have a look at workers, could you tell me more about that "Jupyter kernel messaging"
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Hm, I've tried to make a pyodide package for something with a C library, like so: https://doggieworld.liftm.de/test/pyodide/mecab/mecab.txz . Problem is: it seems to throw a number as an exception, like here https://doggieworld.liftm.de/test/pyodide/mecab/wwmecab.html and as soon as it enters the C++ parts. How do I debug that kind of thing?
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Hm, I guess I'm getting the static linking of a dynamic library part wrong.. it tries to (import "env" "__ZN5MeCab11createModelEPKc" (func (;63;) (type 3))), but that should be internally defined.
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Ohugh. So the package setup.py build a static library and links that. Then pyodide tries to replay the build commands with emscripten, but doesn't replay the "build the library" part.
matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ part was my fault, because I tried to build that library with skip_host and emconfigure, which bypasses the symlinks of pywasmcross. but it also fails because the build command capture opens build.log in the current working directory, and that changes during compiling...
kenjiejima
@kenjiejima

Hi, my name is Kenji, an engineer at brilliant.org.

I started using pyodide a bit, and it's looking pretty exciting.
I realized that scipy is pretty heavy and I wanted to see if I can do something.
The issue is explained here by rth, with a solution proposal from navytux:

iodide-project/pyodide#240

So far, the things I've played around are:

(a) split scipy.data into smaller chunks

It looks like Google Chrome does not cache files over a certain size, and scipy.data
is above the threshold. To force the cache, I tried separating scipy.data into smaller
chunks and tweak the fetchRemotePackage() function in scipy.js file to concatinate them.
This helps the cache situation, but the first access will still remain to be heavy.

(b) remove some scipy subpackages

Delete a few config.add_subpackage() lines in scipy/setup.py and build a custom scipy
to see how much it helps reduce the size of scipy.data. This reduces the size but scipy
still remains to be pretty large. I think this is due to LAPACK and BLAS.

(c) play around with DSO

I tried playing around with DSO -> DSO linking in emscripten. I'm not familiar enough
but at least started to play with it.

So, the next steps I want to try is to dynamically link LAPACK and BLAS. I am assuming
LAPACK is at pyodide/CLAPACK/CLAPACK-WA/lapack_WA.bc as an archive and that gets
statically linked and placed in scipy.data. Building scipy isn't done through a
typical Makefile so it's a bit hard for me to track down how things are built, as I'm
not familiar with it.

May I ask for pointers or ideas of where I can look at to make dynamically linking work?

matrixbot
@matrixbot

ˈt͡sɛːzaɐ̯ The magic all happens in pywasmcross It

  • Symlinks itself to be invoked instead of the normal compilers (cc, ld, ..)
  • Invokes the packages normal setup.py and captures all invocations to cc/ld/.. in a log
  • Replays the invocations, but this time with emcc/emld instead.
  • Invokes setup.py install --skip-build. The output of that is packaged into the .data file by buildpkg.py

Note that scipy has various patches that get applied before build: https://github.com/iodide-project/pyodide/tree/master/packages/scipy/patches

matrixbot
@matrixbot
ˈt͡sɛːzaɐ̯ Ugh. Reading through all of iodide-project/pyodide#240 and a bit of emscripten-core/emscripten#7512 it seems like this will be a lot of fun.. you'll have to change all those static linking hacks into -s RUNTIME_LINKED_LIBS hacks.. いやだいやだ。
kenjiejima
@kenjiejima

LoL. you understand Japanese!
Thanks for the explanation. I looked at the build.log file and got an understanding of what you meant. I was looking at the pyodide/packages/scipy/build/scipy-0.17.1/scipy/linalg/setup.py file, and noticed that deleting a config.add_extension('_flapack', ...) call will remove one static link. Of course that means things LAPACK won't work, but that gave me a hint on how things are built. (I'm basically trying to understand things by poking around)

May I ask what the static linking hacks you mentioned are? (or where I can find one or two)

kenjiejima
@kenjiejima
awesome! thanks a lot. I'll be poking around that area.
Frithjof
@frthjf
Hi everyone, first of all, thanks for this project, really exiting work! I'm wondering if it would be possible to embed the Iodide notebook interface in an existing Javascript application? I have a Javascript SPA dashboard that provides data from a GraphQL backend to the user. It would be great if I could just embed Iodide as a 'component' in the dashboard that would allow advanced users to just script how the data gets represented (say a plot of some tabular data coming from the backend if the dashbaord does not provide such a plot etc).
bcolloran
@bcolloran
hi @frthjf, Iodide is a fairly complex client/server application, so embedding it in another page is probably somewhat difficult, but given the power and flexibility of the web, it's probably possible. you're definitely welcome to give it a shot, and we'd love to hear how it goes!
Frithjof
@frthjf
Hi @bcolloran, thanks, sounds good!
Frithjof
@frthjf
I have a question regarding building your own Python package in pyodide. I've followed the instructions in the docs under 'Creating a Pyodide package' and it works for a pure Python package. However, when I try to build a PyPI package that has dependencies, it fails to compile dependent packages. Is there a way to instruct pyodide to skip the compilation of dependecies? I've tried specifying requirements.run in the meta.yaml as well as applying a patch that removes the dependencies from the setup.py, but no luck so far. How are the package dependencies resolved during the build process and it is possible to skip dependency compilation?
Michael Jurasovic
@jurasofish
@frthjf if you're looking to embed a pyodide component then my pyWeb might be similar to what you're after https://github.com/jurasofish/pyweb
Roman Yurchak
@rth

However, when I try to build a PyPI package that has dependencies, it fails to compile dependent packages. Is there a way to instruct pyodide to skip the compilation of dependecies?

@frthjf Could you please open an issue about it with the detailed log? As far I understand it would run setup.py install and at this point dependencies should already have been installed. If it's not the case, something might indeed be wrong with the dependency resolution.

@jurasofish Nice project! Maybe we could have a section on related projects somewhere in the documentation...

1 reply
William Lachance
@wlach
@rth experimenting with setting up cloudflare with your domain...
(going to use a fresh iodide domain, so no worries about it being public yet)
William Lachance
@wlach
@rth hmm can't setup cloudflare without switching to their nameservers-- this is probably fine, but I should check with brendan and mike before doing so
Roman Yurchak
@rth
@wlach Yeah, it's annoying about having to switch to their NS. Maybe there is a way e.g. to setup cloudflare on one domain then point iodide.io subdomain to it (with some appropriate setting in cloudflare), but does make things a bit more complex (even assuming it would work). Also S3 bucket has to be named exactly as the domain name that points to it apparently, and I'm not sure if it could be made to work with a different domain. So overall changing NS might still be the easiest solution.
William Lachance
@wlach
@rth I think it's fine to change NS, honestly I have better feelings about cloudflare than google
Roman Yurchak
@rth
@wlach True, it shouldn't be too much work in any case.
Frithjof
@frthjf

@frthjf if you're looking to embed a pyodide component then my pyWeb might be similar to what you're after https://github.com/jurasofish/pyweb

Very cool, this looks like a good starting point, thank you!

Ben Goldhaber
@ghabs
Anyone used the scipy.integrate library in pyodide? I've encountered an error: output = _odepack.odeint(func, y0, t, args, Dfun, col_deriv, ml, mu,
pyodide.asm.js:8 AttributeError: 'NoneType' object has no attribute 'odeint' - it seems that _odepack is not available in pyodide?
Roman Yurchak
@rth
@ghabs Yes, scipy is only partially supported for now and is still quite experimental.
Dmitry Mikushin
@dmikushin
Dear Iodide fellows, great work! Pyodide is a very bare-metal thing, according to the docs. So I'm still wondering if there is an existing setup to embed https://alpha.iodide.io/notebooks/34/? kind of interface into a HTML document without a server side?
matrixbot
@matrixbot
jcaesar rth: Btw, is there anything missing for merging https://github.com/iodide-project/pyodide/pull/659#issuecomment-630549846 ?
William Lachance
@wlach
Roman Yurchak
@rth
@matrixbot No that PR is good, we just need to push the corresponding docker image and use it in CI. I'll look into it. Don't hesitate to ping with @ otherwise I don't see the message..
matrixbot
@matrixbot
jcaesar
Oh, woops. My brain was in irc mode.
Frithjof
@frthjf
I'm trying to build the pyyaml package but the emcc step fails as it cannot find the yaml.h header. I believe I've installed all packages required but I'm unsure how to pass in some -lyaml-cpp flag so the compiler can find the header.
Jan Max Meyer
@phorward
Hi there, I would like to ask without making an issue: Are there yet any efforts in trying to bring Python 3.8 into Pyodide?
Roman Yurchak
@rth
@phorward Not as far as I'm aware. It would be certainly very useful to try. For porting patches, it might be helpful put patches as commits in a CPython branch from 3.7.4 and try to rebase on top of 3.8.0 and re-extract patches as files.
Roman Yurchak
@rth
Also BTW, I have submitted a sprint proposal for pyodide for the (virtual) scipy 2020 conference (https://www.scipy2020.scipy.org/) which would happen for sprints on July 11-12 weekend. I haven't heard back, but I imagine it should be fine. If anyone is interested in participating you need to register on the scipy website (sprints registrations is free, but registration is necessary).