Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 14 22:41
    Travis dropbox/pyston (master) errored (4687)
  • Dec 27 2019 23:49
  • Jan 17 2019 05:30
    xxllp commented #1426
  • Dec 13 2018 23:14
    deronnax closed #1421
  • Jul 14 2018 08:28
    corporatepiyush commented #1426
  • May 23 2018 21:37
    toshok closed #528
  • May 23 2018 21:37
    toshok closed #558
  • May 23 2018 21:34
    toshok closed #609
  • May 23 2018 21:34
    toshok closed #756
  • Feb 21 2018 05:25
    osman-masood commented #1426
  • Feb 03 2018 02:34
    skywind3000 commented #1426
  • Feb 01 2018 22:06
    danluu commented #1426
  • Jan 27 2018 19:57
    lesshaste commented #1426
  • Jan 12 2018 13:11
    skywind3000 opened #1426
  • Nov 12 2017 19:59
    xoviat commented #1421
  • Aug 25 2017 15:58
    Travis dropbox/pyston (py3) passed (4685)
  • Aug 25 2017 15:31
    kmod commented #1425
  • Aug 25 2017 15:31
    kmod closed #1425
  • Aug 24 2017 15:02
    kmod commented #1422
  • Aug 01 2017 17:06
    denji closed #378
Sun
@Daetalus
great!
Stefan O'Neil
@stefanoneil
Thanks for the helpful pointers @kmod !
Stefan O'Neil
@stefanoneil
I have another quick question for you or anyone who is decently familiar with the Pyston source code. Something I am working on in my current project is extending Pyston grammar with an additional keyword. In CPython, Python/graminit.c and Include/graminit.h are auto-generated from Grammar/Grammar when you run make.
In tracking down all the files I need to modify to implement an additional keyword in Pyston, I can’t seem to find the grammar file from which graminit.c and graminit.h are generated. The only grammar file I can find is from_cpython/Lib/lib2to3/Grammar.txt, but modifying this does not seem to trigger a regeneration of the graminit files.
Where should I be looking for the grammar file that Pyston generates these from?
Sun
@Daetalus
Here: https://github.com/dropbox/pyston/tree/master/src/codegen. But first you may need copy new generated CPython Python-ast.h file. And then extend the Pyston interpreter and jit.
Stefan O'Neil
@stefanoneil
Excellent, thanks a ton for that
I believe in CPython, Python-ast.h and Python-ast.c are both generated by Parser/asdl_c.py whenever Parser/Python.asdl is changed. Do you happen to know if Pyston works this way as well?
Stefan O'Neil
@stefanoneil
Also, you linked me the src/codegen directory, but I am not seeing a grammar specification file in here. Does that mean Pyston not use a grammar specification file similar to CPython's Grammar/Grammar, but rather generates graminit.c and graminit.h some other way?
Again, I really appreciate all the help
Sun
@Daetalus
Will reply you later...
Stefan O'Neil
@stefanoneil
Sure thing, thanks again
Kevin Modzelewski
@kmod
I believe we copied Python-ast.h and Python-ast.c in from the CPython repository
and probably graminit.h/c as well
We never experimented with changing the grammar, but maybe one way to do that is to make the change in a CPython checkout, regenerate the various files, and re-copy them into Pyston
you probably need to regenerate src/codegen/cpython_ast.cpp (it includes the script to do so at the top of the file)
as I said, we haven't tried this, so I don't expect it to be super straightforward. let us know if you run into issues!
Stefan O'Neil
@stefanoneil
Oh I see, that’s good to know, thanks.
Stefan O'Neil
@stefanoneil
Yup, just confirmed, the graminit files are indeed identical with CPython. My apologies for the confusion.
Stefan O'Neil
@stefanoneil
Taking your recommendation for regenerating the graminit files, plus modifying a number of other files, I have successfully extended Pyston with a keyword, except for one problem
if I try to use the keyword from within a function, Pyston crashes.
in every other context, it seems to work just fine
the behavior of the keyword itself shouldn't be breaking anything, since it is just a trivial keyword for now
therefore i'm thinking maybe i need to modify Pyston's symbol table
Stefan O'Neil
@stefanoneil
in CPython the symbol table is mostly managed by Python/symtable.c, which isn't in Pyston
would src/codegen/irgen.cpp be the right place to look for this?
Kevin Modzelewski
@kmod
We have three different ways that we execute Python code; irgen.cpp is the third, LLVM compilation
I'd pass -I to the pyston process, which forces it to stay in the AST interpreter
and get that working first
-I will force it to stay in codegen/ast_interpreter.cpp and make things much simpler
if that doesn't help, it might be some of the static analysis we do; if you post the crash then I could make a more informed guess
Stefan O'Neil
@stefanoneil
Sure thing:
$ ./pyston_dbg -I
Pyston v0.7.0 (rev f7e8fcf7d57c624da5ac8a8be7d0d55c260ff32e), targeting Python 2.7.8
>> 
>> 
>> escal x


reached an escal statement

>> 
>> 
>> def foo():
...  escal y
...  return 4
... 
../../src/core/ast.h:1151: virtual bool pyston::ASTVisitor::visit_escal(pyston::AST_Escal *): Assertion `0' failed: 
Someone called abort!
Aborted (core dumped)
I also have gdb output, in case it is more informative:
(gdb) run happytest.py
Starting program: /home/escal3/debug_pyston/pyston/pyston_dbg happytest.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".


reached an escal statement

../../src/core/ast.h:1151: virtual bool pyston::ASTVisitor::visit_escal(pyston::AST_Escal *): Assertion `0' failed: 
Someone called abort!

Program received signal SIGABRT, Aborted.
0x00007ffff5185c37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56    ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  0x00007ffff5185c37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff5189028 in __GI_abort () at abort.c:89
#2  0x00000000007d0245 in abort () at ../../src/codegen/unwinding.cpp:1158
#3  0x00000000006167b7 in pyston::ASTVisitor::visit_escal (this=0x7fffffffaa88, 
    node=0x21f22d8) at ../../src/core/ast.h:1151
#4  0x00000000007dd1d1 in pyston::AST_Escal::accept (this=0x21f22d8, 
    v=0x7fffffffaa88) at ../../src/core/ast.cpp:557
#5  0x00000000014afdbc in pyston::NameCollectorVisitor::visit_functiondef (
    this=0x7fffffffaa88, node=0x21f2190)
    at ../../src/analysis/scoping_analysis.cpp:683
#6  0x00000000014e4ca5 in pyston::AST_FunctionDef::accept (this=0x21f2190, 
    v=0x7fffffffaa88) at ../../src/core/ast.cpp:521
#7  0x00000000014afcdb in pyston::NameCollectorVisitor::collect (node=0x21f2190, 
    map=0x7fffffffab18, scoping=0x7fffffffc3b0)
    at ../../src/analysis/scoping_analysis.cpp:795
#8  0x0000000000612f9e in pyston::ScopingAnalysis::analyzeSubtree (
    this=0x7fffffffc3b0, node=0x21f2190)
    at ../../src/analysis/scoping_analysis.cpp:977
#9  0x00000000014b0e85 in pyston::ScopingAnalysis::getScopeInfoForNode (
    this=0x7fffffffc3b0, node=0x21f2190)
    at ../../src/analysis/scoping_analysis.cpp:986
#10 0x00000000014f3142 in pyston::ModuleCFGProcessor::runRecursively (
    this=0x7fffffffc3b0, body=..., name=0x7ffff49af2e0, lineno=3, args=0x21f21f0, 
    orig_node=0x21f2190) at ../../src/core/cfg.cpp:3538
#11 0x00000000014f4271 in pyston::CFGVisitor::visit_functiondef (
    this=0x7fffffffbba8, node=0x21f2190) at ../../src/core/cfg.cpp:1966
#12 0x00000000014e4ca5 in pyston::AST_FunctionDef::accept (this=0x21f2190, 
    v=0x7fffffffbba8) at ../../src/core/ast.cpp:521
#13 0x00000000007f217f in pyston::computeCFG (body=..., 
    ast_type=pyston::AST_TYPE::Module, lineno=1, args=0x0, filename=0x7ffff49adf48, 
    source=0x251de40, param_names=..., scoping=0x23ed9c0, cfgizer=0x7fffffffc3b0)
    at ../../src/core/cfg.cpp:3380
#14 0x00000000014f36ec in pyston::ModuleCFGProcessor::runRecursively (
    this=0x7fffffffc3b0, body=..., name=0x7ffff7e8f2b0, lineno=1, args=0x0, 
    orig_node=0x21f20a0) at ../../src/core/cfg.cpp:3581
#15 0x00000000014f3d3f in pyston::computeAllCFGs (ast=0x21f20a0, 
    globals_from_module=true, future_flags=16, fn=0x7ffff49adf48, bm=0x23686e0)
    at ../../src/core/cfg.cpp:3597
#16 0x0000000000752df2 in pyston::compileAndRunModule (m=0x21f20a0, bm=0x23686e0)
    at ../../src/codegen/irgen/hooks.cpp:234
#17 0x00000000005fa84b in pyston::main (argc=2, argv=0x7fffffffdfc8)
    at ../../src/jit.cpp:536
#18 0x00000000005f95e2 in main (argc=2, argv=0x7fffffffdfc8)
    at ../../src/jit.cpp:591
Kevin Modzelewski
@kmod
Is that your new keyword?
Sounds like it just doesn't know what to do with it
Stefan O'Neil
@stefanoneil
my apologies for the late reply, i've been away
you were right about it being the scoping analysis
there were some places in src/analysis/scoping_analysis.cppwhere I didn't realize I needed to make additions
hence why the crash only showed up in function definitions
Stefan O'Neil
@stefanoneil
oh and yes, "escal" is just the name I am using for the keyword right now
Kevin Modzelewski
@kmod
Ah, yeah there are some differences between how we handle functions and modules, which is the source of some annoying quirks like this
Sun
@Daetalus
Hi, @stefanoneil you are working on a private repo?
Ali Ahsan
@aliahsan07
what are the instructions of running pyston after entering the docker?
Ali Ahsan
@aliahsan07
wow this project is effectively dead
Kevin Modzelewski
@kmod
yes unfortunately
When in the docker container, python will end up running Pyston
And you can pip install things as well
Ali Ahsan
@aliahsan07
@kmod can pyston be used to develop some sort of a control flow graph, given a python program?