These are chat archives for dropbox/pyston

9th
Oct 2017
Stefan O'Neil
@stefanoneil
Oct 09 2017 01:56
Thanks for the helpful pointers @kmod !
Stefan O'Neil
@stefanoneil
Oct 09 2017 02:05
I have another quick question for you or anyone who is decently familiar with the Pyston source code. Something I am working on in my current project is extending Pyston grammar with an additional keyword. In CPython, Python/graminit.c and Include/graminit.h are auto-generated from Grammar/Grammar when you run make.
In tracking down all the files I need to modify to implement an additional keyword in Pyston, I can’t seem to find the grammar file from which graminit.c and graminit.h are generated. The only grammar file I can find is from_cpython/Lib/lib2to3/Grammar.txt, but modifying this does not seem to trigger a regeneration of the graminit files.
Where should I be looking for the grammar file that Pyston generates these from?
Sun
@Daetalus
Oct 09 2017 02:11
Here: https://github.com/dropbox/pyston/tree/master/src/codegen. But first you may need copy new generated CPython Python-ast.h file. And then extend the Pyston interpreter and jit.
Stefan O'Neil
@stefanoneil
Oct 09 2017 02:32
Excellent, thanks a ton for that
I believe in CPython, Python-ast.h and Python-ast.c are both generated by Parser/asdl_c.py whenever Parser/Python.asdl is changed. Do you happen to know if Pyston works this way as well?
Stefan O'Neil
@stefanoneil
Oct 09 2017 02:43
Also, you linked me the src/codegen directory, but I am not seeing a grammar specification file in here. Does that mean Pyston not use a grammar specification file similar to CPython's Grammar/Grammar, but rather generates graminit.c and graminit.h some other way?
Again, I really appreciate all the help
Sun
@Daetalus
Oct 09 2017 02:47
Will reply you later...
Stefan O'Neil
@stefanoneil
Oct 09 2017 02:47
Sure thing, thanks again
Kevin Modzelewski
@kmod
Oct 09 2017 16:25
I believe we copied Python-ast.h and Python-ast.c in from the CPython repository
and probably graminit.h/c as well
We never experimented with changing the grammar, but maybe one way to do that is to make the change in a CPython checkout, regenerate the various files, and re-copy them into Pyston
you probably need to regenerate src/codegen/cpython_ast.cpp (it includes the script to do so at the top of the file)
as I said, we haven't tried this, so I don't expect it to be super straightforward. let us know if you run into issues!
Stefan O'Neil
@stefanoneil
Oct 09 2017 16:53
Oh I see, that’s good to know, thanks.
Stefan O'Neil
@stefanoneil
Oct 09 2017 17:06
Yup, just confirmed, the graminit files are indeed identical with CPython. My apologies for the confusion.
Stefan O'Neil
@stefanoneil
Oct 09 2017 17:33
Taking your recommendation for regenerating the graminit files, plus modifying a number of other files, I have successfully extended Pyston with a keyword, except for one problem
if I try to use the keyword from within a function, Pyston crashes.
in every other context, it seems to work just fine
the behavior of the keyword itself shouldn't be breaking anything, since it is just a trivial keyword for now
therefore i'm thinking maybe i need to modify Pyston's symbol table
Stefan O'Neil
@stefanoneil
Oct 09 2017 17:45
in CPython the symbol table is mostly managed by Python/symtable.c, which isn't in Pyston
would src/codegen/irgen.cpp be the right place to look for this?
Kevin Modzelewski
@kmod
Oct 09 2017 18:27
We have three different ways that we execute Python code; irgen.cpp is the third, LLVM compilation
I'd pass -I to the pyston process, which forces it to stay in the AST interpreter
and get that working first
-I will force it to stay in codegen/ast_interpreter.cpp and make things much simpler
if that doesn't help, it might be some of the static analysis we do; if you post the crash then I could make a more informed guess
Stefan O'Neil
@stefanoneil
Oct 09 2017 18:38
Sure thing:
$ ./pyston_dbg -I
Pyston v0.7.0 (rev f7e8fcf7d57c624da5ac8a8be7d0d55c260ff32e), targeting Python 2.7.8
>> 
>> 
>> escal x


reached an escal statement

>> 
>> 
>> def foo():
...  escal y
...  return 4
... 
../../src/core/ast.h:1151: virtual bool pyston::ASTVisitor::visit_escal(pyston::AST_Escal *): Assertion `0' failed: 
Someone called abort!
Aborted (core dumped)
I also have gdb output, in case it is more informative:
(gdb) run happytest.py
Starting program: /home/escal3/debug_pyston/pyston/pyston_dbg happytest.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".


reached an escal statement

../../src/core/ast.h:1151: virtual bool pyston::ASTVisitor::visit_escal(pyston::AST_Escal *): Assertion `0' failed: 
Someone called abort!

Program received signal SIGABRT, Aborted.
0x00007ffff5185c37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56    ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  0x00007ffff5185c37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff5189028 in __GI_abort () at abort.c:89
#2  0x00000000007d0245 in abort () at ../../src/codegen/unwinding.cpp:1158
#3  0x00000000006167b7 in pyston::ASTVisitor::visit_escal (this=0x7fffffffaa88, 
    node=0x21f22d8) at ../../src/core/ast.h:1151
#4  0x00000000007dd1d1 in pyston::AST_Escal::accept (this=0x21f22d8, 
    v=0x7fffffffaa88) at ../../src/core/ast.cpp:557
#5  0x00000000014afdbc in pyston::NameCollectorVisitor::visit_functiondef (
    this=0x7fffffffaa88, node=0x21f2190)
    at ../../src/analysis/scoping_analysis.cpp:683
#6  0x00000000014e4ca5 in pyston::AST_FunctionDef::accept (this=0x21f2190, 
    v=0x7fffffffaa88) at ../../src/core/ast.cpp:521
#7  0x00000000014afcdb in pyston::NameCollectorVisitor::collect (node=0x21f2190, 
    map=0x7fffffffab18, scoping=0x7fffffffc3b0)
    at ../../src/analysis/scoping_analysis.cpp:795
#8  0x0000000000612f9e in pyston::ScopingAnalysis::analyzeSubtree (
    this=0x7fffffffc3b0, node=0x21f2190)
    at ../../src/analysis/scoping_analysis.cpp:977
#9  0x00000000014b0e85 in pyston::ScopingAnalysis::getScopeInfoForNode (
    this=0x7fffffffc3b0, node=0x21f2190)
    at ../../src/analysis/scoping_analysis.cpp:986
#10 0x00000000014f3142 in pyston::ModuleCFGProcessor::runRecursively (
    this=0x7fffffffc3b0, body=..., name=0x7ffff49af2e0, lineno=3, args=0x21f21f0, 
    orig_node=0x21f2190) at ../../src/core/cfg.cpp:3538
#11 0x00000000014f4271 in pyston::CFGVisitor::visit_functiondef (
    this=0x7fffffffbba8, node=0x21f2190) at ../../src/core/cfg.cpp:1966
#12 0x00000000014e4ca5 in pyston::AST_FunctionDef::accept (this=0x21f2190, 
    v=0x7fffffffbba8) at ../../src/core/ast.cpp:521
#13 0x00000000007f217f in pyston::computeCFG (body=..., 
    ast_type=pyston::AST_TYPE::Module, lineno=1, args=0x0, filename=0x7ffff49adf48, 
    source=0x251de40, param_names=..., scoping=0x23ed9c0, cfgizer=0x7fffffffc3b0)
    at ../../src/core/cfg.cpp:3380
#14 0x00000000014f36ec in pyston::ModuleCFGProcessor::runRecursively (
    this=0x7fffffffc3b0, body=..., name=0x7ffff7e8f2b0, lineno=1, args=0x0, 
    orig_node=0x21f20a0) at ../../src/core/cfg.cpp:3581
#15 0x00000000014f3d3f in pyston::computeAllCFGs (ast=0x21f20a0, 
    globals_from_module=true, future_flags=16, fn=0x7ffff49adf48, bm=0x23686e0)
    at ../../src/core/cfg.cpp:3597
#16 0x0000000000752df2 in pyston::compileAndRunModule (m=0x21f20a0, bm=0x23686e0)
    at ../../src/codegen/irgen/hooks.cpp:234
#17 0x00000000005fa84b in pyston::main (argc=2, argv=0x7fffffffdfc8)
    at ../../src/jit.cpp:536
#18 0x00000000005f95e2 in main (argc=2, argv=0x7fffffffdfc8)
    at ../../src/jit.cpp:591