dropbox/pyston

An open-source Python implementation using JIT techniques.

Kevin Modzelewski

@kmod

We have three different ways that we execute Python code; irgen.cpp is the third, LLVM compilation

I'd pass -I to the pyston process, which forces it to stay in the AST interpreter

and get that working first

-I will force it to stay in codegen/ast_interpreter.cpp and make things much simpler

if that doesn't help, it might be some of the static analysis we do; if you post the crash then I could make a more informed guess

Stefan O'Neil

@stefanoneil

Sure thing:

$ ./pyston_dbg -I
Pyston v0.7.0 (rev f7e8fcf7d57c624da5ac8a8be7d0d55c260ff32e), targeting Python 2.7.8
>> 
>> 
>> escal x


reached an escal statement

>> 
>> 
>> def foo():
...  escal y
...  return 4
... 
../../src/core/ast.h:1151: virtual bool pyston::ASTVisitor::visit_escal(pyston::AST_Escal *): Assertion `0' failed: 
Someone called abort!
Aborted (core dumped)

I also have gdb output, in case it is more informative:

(gdb) run happytest.py
Starting program: /home/escal3/debug_pyston/pyston/pyston_dbg happytest.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".


reached an escal statement

../../src/core/ast.h:1151: virtual bool pyston::ASTVisitor::visit_escal(pyston::AST_Escal *): Assertion `0' failed: 
Someone called abort!

Program received signal SIGABRT, Aborted.
0x00007ffff5185c37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56    ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  0x00007ffff5185c37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff5189028 in __GI_abort () at abort.c:89
#2  0x00000000007d0245 in abort () at ../../src/codegen/unwinding.cpp:1158
#3  0x00000000006167b7 in pyston::ASTVisitor::visit_escal (this=0x7fffffffaa88, 
    node=0x21f22d8) at ../../src/core/ast.h:1151
#4  0x00000000007dd1d1 in pyston::AST_Escal::accept (this=0x21f22d8, 
    v=0x7fffffffaa88) at ../../src/core/ast.cpp:557
#5  0x00000000014afdbc in pyston::NameCollectorVisitor::visit_functiondef (
    this=0x7fffffffaa88, node=0x21f2190)
    at ../../src/analysis/scoping_analysis.cpp:683
#6  0x00000000014e4ca5 in pyston::AST_FunctionDef::accept (this=0x21f2190, 
    v=0x7fffffffaa88) at ../../src/core/ast.cpp:521
#7  0x00000000014afcdb in pyston::NameCollectorVisitor::collect (node=0x21f2190, 
    map=0x7fffffffab18, scoping=0x7fffffffc3b0)
    at ../../src/analysis/scoping_analysis.cpp:795
#8  0x0000000000612f9e in pyston::ScopingAnalysis::analyzeSubtree (
    this=0x7fffffffc3b0, node=0x21f2190)
    at ../../src/analysis/scoping_analysis.cpp:977
#9  0x00000000014b0e85 in pyston::ScopingAnalysis::getScopeInfoForNode (
    this=0x7fffffffc3b0, node=0x21f2190)
    at ../../src/analysis/scoping_analysis.cpp:986
#10 0x00000000014f3142 in pyston::ModuleCFGProcessor::runRecursively (
    this=0x7fffffffc3b0, body=..., name=0x7ffff49af2e0, lineno=3, args=0x21f21f0, 
    orig_node=0x21f2190) at ../../src/core/cfg.cpp:3538
#11 0x00000000014f4271 in pyston::CFGVisitor::visit_functiondef (
    this=0x7fffffffbba8, node=0x21f2190) at ../../src/core/cfg.cpp:1966
#12 0x00000000014e4ca5 in pyston::AST_FunctionDef::accept (this=0x21f2190, 
    v=0x7fffffffbba8) at ../../src/core/ast.cpp:521
#13 0x00000000007f217f in pyston::computeCFG (body=..., 
    ast_type=pyston::AST_TYPE::Module, lineno=1, args=0x0, filename=0x7ffff49adf48, 
    source=0x251de40, param_names=..., scoping=0x23ed9c0, cfgizer=0x7fffffffc3b0)
    at ../../src/core/cfg.cpp:3380
#14 0x00000000014f36ec in pyston::ModuleCFGProcessor::runRecursively (
    this=0x7fffffffc3b0, body=..., name=0x7ffff7e8f2b0, lineno=1, args=0x0, 
    orig_node=0x21f20a0) at ../../src/core/cfg.cpp:3581
#15 0x00000000014f3d3f in pyston::computeAllCFGs (ast=0x21f20a0, 
    globals_from_module=true, future_flags=16, fn=0x7ffff49adf48, bm=0x23686e0)
    at ../../src/core/cfg.cpp:3597
#16 0x0000000000752df2 in pyston::compileAndRunModule (m=0x21f20a0, bm=0x23686e0)
    at ../../src/codegen/irgen/hooks.cpp:234
#17 0x00000000005fa84b in pyston::main (argc=2, argv=0x7fffffffdfc8)
    at ../../src/jit.cpp:536
#18 0x00000000005f95e2 in main (argc=2, argv=0x7fffffffdfc8)
    at ../../src/jit.cpp:591

Kevin Modzelewski

@kmod

Is that your new keyword?

Sounds like it just doesn't know what to do with it

Stefan O'Neil

@stefanoneil

my apologies for the late reply, i've been away

you were right about it being the scoping analysis

there were some places in src/analysis/scoping_analysis.cppwhere I didn't realize I needed to make additions

hence why the crash only showed up in function definitions

Stefan O'Neil

@stefanoneil

oh and yes, "escal" is just the name I am using for the keyword right now

Kevin Modzelewski

@kmod

Ah, yeah there are some differences between how we handle functions and modules, which is the source of some annoying quirks like this

Sun

@Daetalus

Hi, @stefanoneil you are working on a private repo?

Ali Ahsan

@aliahsan07

what are the instructions of running pyston after entering the docker?

Ali Ahsan

@aliahsan07

wow this project is effectively dead

Kevin Modzelewski

@kmod

yes unfortunately

When in the docker container, python will end up running Pyston

And you can pip install things as well

Ali Ahsan

@aliahsan07

@kmod can pyston be used to develop some sort of a control flow graph, given a python program?

Kevin Modzelewski

@kmod

Only inside of a single function

salvator lopez

@salvator_lopez_twitter

what about guido van rossum ? .. is he working on dropbox when u switched to Golang and Dont wanna Develop Pyston anyway ?

Tong

@tongzhou_gitlab

hi, is Pyston still under development?

our lab needs a LLVM based Python compiler, so far I only found Pyston and Numba

Kevin Modzelewski

@kmod

Nope, not under development

Python isn't really compile-able, so I think what you're asking for isn't a fit for Python

Probably the closest you can get is Cython->clang

Tong

@tongzhou_gitlab

hmm okay

thanks for the info!

Pratyush Das

@reikdas

Is anyone aware of any forks where people have continued working on this? (and still are)

Kevin Modzelewski

@kmod

I may have an as-yet-unannounced successor project in the works :)

Pratyush Das

@reikdas

Sun

@Daetalus

@kmod Hi, it's your personal project?

Seo Sanghyeon

@sanxiyn

Also: is it related to kmod/dpro?

Kevin Modzelewski

@kmod

@sanxiyn yep you found it!

It's still in the very early stages, but Marius and I have been working on expanding some of the ideas from Pyston into their own project

Kevin Modzelewski

@kmod

I'll post about it in more detail when we have more to show, but until then feel free to take a look / ask questions!

AN Long

@aisk

Cool!

AN Long

@aisk

Dose the optimized codes will be working on the stuff which pep523 described, thus it can be support by CPython "officially" and the user can using it without any capalibility corcerns?

Kevin Modzelewski

@kmod

We use Cython, which is a different way of achieving CPython support

There are some small compatibility differences but for the most part it's just drop-in :)

AN Long

@aisk

Just have a little experience on cython, but looks like cython can not aot compile the whole standard library (or runtime). So if the new project is based on cython, does it have the ability to optimize the runtime?

Kevin Modzelewski

@kmod

Oh interesting! Do you have any more info on this?

I've only seen https://cython.readthedocs.io/en/latest/src/userguide/limitations.html which makes their compatibility seem quite good

Also I'd say we "use" cython and are not "based on" cython -- we use cython to lower Python code to C code, and then do a number of C-level tricks to optimize the generated code

I'm not sure which "runtime" you're referring to, but we definitely optimize the CPython C runtime :)

Where communities thrive

People

Repo info

Activity