dropbox/pyston

An open-source Python implementation using JIT techniques.

Sun

@Daetalus

Hi, @stefanoneil you are working on a private repo?

Ali Ahsan

@aliahsan07

what are the instructions of running pyston after entering the docker?

Ali Ahsan

@aliahsan07

wow this project is effectively dead

Kevin Modzelewski

@kmod

yes unfortunately

When in the docker container, python will end up running Pyston

And you can pip install things as well

Ali Ahsan

@aliahsan07

@kmod can pyston be used to develop some sort of a control flow graph, given a python program?

Kevin Modzelewski

@kmod

Only inside of a single function

salvator lopez

@salvator_lopez_twitter

what about guido van rossum ? .. is he working on dropbox when u switched to Golang and Dont wanna Develop Pyston anyway ?

Tong

@tongzhou_gitlab

hi, is Pyston still under development?

our lab needs a LLVM based Python compiler, so far I only found Pyston and Numba

Kevin Modzelewski

@kmod

Nope, not under development

Python isn't really compile-able, so I think what you're asking for isn't a fit for Python

Probably the closest you can get is Cython->clang

Tong

@tongzhou_gitlab

hmm okay

thanks for the info!

Pratyush Das

@reikdas

Is anyone aware of any forks where people have continued working on this? (and still are)

Kevin Modzelewski

@kmod

I may have an as-yet-unannounced successor project in the works :)

Pratyush Das

@reikdas

Sun

@Daetalus

@kmod Hi, it's your personal project?

Seo Sanghyeon

@sanxiyn

Also: is it related to kmod/dpro?

Kevin Modzelewski

@kmod

@sanxiyn yep you found it!

It's still in the very early stages, but Marius and I have been working on expanding some of the ideas from Pyston into their own project

Kevin Modzelewski

@kmod

I'll post about it in more detail when we have more to show, but until then feel free to take a look / ask questions!

An Long

@aisk

Cool!

An Long

@aisk

Dose the optimized codes will be working on the stuff which pep523 described, thus it can be support by CPython "officially" and the user can using it without any capalibility corcerns?

Kevin Modzelewski

@kmod

We use Cython, which is a different way of achieving CPython support

There are some small compatibility differences but for the most part it's just drop-in :)

An Long

@aisk

Just have a little experience on cython, but looks like cython can not aot compile the whole standard library (or runtime). So if the new project is based on cython, does it have the ability to optimize the runtime?

Kevin Modzelewski

@kmod

Oh interesting! Do you have any more info on this?

I've only seen https://cython.readthedocs.io/en/latest/src/userguide/limitations.html which makes their compatibility seem quite good

Also I'd say we "use" cython and are not "based on" cython -- we use cython to lower Python code to C code, and then do a number of C-level tricks to optimize the generated code

I'm not sure which "runtime" you're referring to, but we definitely optimize the CPython C runtime :)

Also if anyone is interested, I started putting up some issues on our issue tracker. They should be pretty approachable and would be helpful!

Sun

@Daetalus

I modified Cython source code for support new syntax, which is add a type called "nogil extension" to get rid of Cython environment, so we can enable multicore support for Python-like syntax. https://lab.nexedi.com/Daetalus/cython/commits/nogil_extension_lwan

Sun

@Daetalus

It is a personal/community project, right?

Kevin Modzelewski

@kmod

Cool!

You mean nitrous (previously called dpro)? Yep a community project :)

An Long

@aisk

Had a little test based on the pyximport described here: https://cython.readthedocs.io/en/latest/src/tutorial/cython_tutorial.html#pyximport-cython-compilation-for-developers

If have a file:

import pyximport
pyximport.install(pyimport=True, build_dir='xx')

import six

Since six is a pure python module, looks like cython can cythonize it totally, but I got an error while run this file:

Traceback (most recent call last):
  File "a.py", line 4, in <module>
    import six
  File "/Users/anlong/Library/Python/2.7/lib/python/site-packages/pyximport/pyximport.py", line 419, in load_module
    return load_module(fullname, source_path, so_path=so_path, is_package=is_package)
  File "/Users/anlong/Library/Python/2.7/lib/python/site-packages/pyximport/pyximport.py", line 233, in load_module
    exec("raise exc, None, tb", {'exc': exc, 'tb': tb})
  File "/Users/anlong/Library/Python/2.7/lib/python/site-packages/pyximport/pyximport.py", line 216, in load_module
    mod = imp.load_dynamic(name, so_path)
  File "six.py", line 805, in init six
    _add_doc(reraise, """Reraise an exception.""")
ImportError: Building module six failed: ["NameError: name 'reraise' is not defined\n"]

An Long

@aisk

A quick dig is that reraise is defined with exec statement with reraise function as a string, so seems cython don't support this use case.

Another try is just change the import six to import requests, with another error:

Traceback (most recent call last):
  File "a.py", line 4, in <module>
    import requests
  File "/Users/anlong/Library/Python/2.7/lib/python/site-packages/pyximport/pyximport.py", line 419, in load_module
    return load_module(fullname, source_path, so_path=so_path, is_package=is_package)
  File "/Users/anlong/Library/Python/2.7/lib/python/site-packages/pyximport/pyximport.py", line 233, in load_module
    exec("raise exc, None, tb", {'exc': exc, 'tb': tb})
  File "/Users/anlong/Library/Python/2.7/lib/python/site-packages/pyximport/pyximport.py", line 216, in load_module
    mod = imp.load_dynamic(name, so_path)
  File "__init__.py", line 43, in init requests.__init__
  File "/Users/anlong/Library/Python/2.7/lib/python/site-packages/pyximport/pyximport.py", line 419, in load_module
    return load_module(fullname, source_path, so_path=so_path, is_package=is_package)
  File "/Users/anlong/Library/Python/2.7/lib/python/site-packages/pyximport/pyximport.py", line 233, in load_module
    exec("raise exc, None, tb", {'exc': exc, 'tb': tb})
  File "/Users/anlong/Library/Python/2.7/lib/python/site-packages/pyximport/pyximport.py", line 216, in load_module
    mod = imp.load_dynamic(name, so_path)
  File "__init__.py", line 8, in init urllib3.__init__
ImportError: Building module requests failed: ["ImportError: Building module urllib3 failed: ['ValueError: Attempted relative import in non-package\\n']\n"]

An Long

@aisk

Another try is with flask and jedi, and they all failed.

An Long

@aisk

Looks like cython have more limitations than the documents said. Improve the compatibility of cython is good, but I don't know how long the gaps were.

Kevin Modzelewski

@kmod

Interesting! A bit disappointing, but good to know

An Long

@aisk

FYI, another cython like tools is nuitka, and looks like it can compile more modules to binary.

Pratyush Das

@reikdas

Might be interesting to read, Guido Van Rossum was rather harsh about nuitka - https://www.pycon.it/conference/talks/nuitka-the-python-compiler

Where communities thrive

People

Repo info

Activity