These are chat archives for dropbox/pyston

29th
Jun 2015
Travis Hance
@tjhance
Jun 29 2015 06:10
hey @toshok I was looking into statically allocating all the builtin type objects to see if it would give an improvement but I’m kinda stuck as far as the GC is concerned. It seems like if I do this then I have to register the types as “nonheap objects” but those are conservatively scanned. What would it take to do precise scanning on objects outside of the global_heap? Is this an easy or hard thing to change?
Ishan Khare
@ishankhare07
Jun 29 2015 07:57
what's the best resource to get started?
Travis Hance
@tjhance
Jun 29 2015 08:13
hm so a decent way to get started is just to pick a failing test that looks interesting and fix it. The cpython tests are probably best for that (in test/cpython/), often you’ll just find that pyston is missing some function or object or attribute or whatever. Then ask here and we can help you find what files to look in and so on.
you can also see if any of our doucmented issues look interesting https://github.com/dropbox/pyston/issues (some are marked ‘probably easy’, hopefully those are probably easy)
Kevin Modzelewski
@kmod
Jun 29 2015 08:40
yeah, failing cpython tests are a good place to start
ie picking something from grep -l 'expected: fail' test/cpython/*.py
Kevin Modzelewski
@kmod
Jun 29 2015 10:03
Ok, I updated some of our documentation on what tests are failing and why
I'd recommend just picking something that sounds easy and taking a crack at it
definitely feel free to ask questions here -- there's not much in the way of formal documentation right now
Chris Toshok
@toshok
Jun 29 2015 17:11
@tjhance it shouldn’t be hard to add that sort of thing. in fact there’s a commit in the incremental-tb-improvements PR that might help: toshok/pyston@a0e9dff
with that you should be able to just do something like: gc::registerRootCallback(rooted_type, typeGCHandler);
Travis Hance
@tjhance
Jun 29 2015 17:14
oh cool thanks
Chris Toshok
@toshok
Jun 29 2015 17:15
although it’s conceivable you’ll run afoul of the gc::isValidGCObject asserts everywhere
since that permits non-heap roots, but not the pointers registered with registerRootCallback
Rudi Chen
@rudi-c
Jun 29 2015 18:01
Rebasing my GC changes over the changes previous ~30 commits or so somehow made a lot of benchmarks that were previously 10% slower <5% slower (in the case of interp2, even 7% faster!). @kmod any idea what might have happened?
Rudi Chen
@rudi-c
Jun 29 2015 18:16
Also I get this error on Travis, gcc only
                   incremental_tb.py    FAILED (Exited with code -6 (expected code 0))
Warning: corrupt or non-Pyston .pyc file found; ignoring
97 10 99 77
/home/travis/build/dropbox/pyston/src/codegen/parser.cpp:1220: pyston::AST_Module* pyston::caching_parse_file(const char*): Assertion `tries <= MAX_TRIES' failed: repeatedly failing to parse file
Someone called abort!
Traceback (most recent call last):
  File "/home/travis/build/dropbox/pyston/test/tests/incremental_tb.py", line 1, in <module>:
    import traceback
  File "/home/travis/pyston-build/from_cpython/Lib/traceback.py", line 6, in <module>:
    import linecache
  File "/home/travis/pyston-build/from_cpython/Lib/linecache.py", line 9, in <module>:
    import os
  File "/home/travis/pyston-build/from_cpython/Lib/os.py", line 49, in <module>:
    import posixpath as path
  File "/home/travis/pyston-build/from_cpython/Lib/posixpath.py", line 17, in <module>:
    import warnings
Chris Toshok
@toshok
Jun 29 2015 19:12
i woke up at stupid-early o’clock this morning and played around with specializing the slowpaths by whether or not they were rewritable
so instead of Box* typeLookup(BoxedClass* cls, llvm::StringRef attr, GetattrRewriteArgs* rewrite_args), I have:
template <bool rewritable>
static Box* typeLookupRewrite(BoxedClass* cls, llvm::StringRef attr, GetattrRewriteArgs* rewrite_args) { … }
and everywhere in typeLookupRewrite where we’d normally do if (rewrite_args), it does if (rewritable && rewrite_args)
passing false for rewritable gives us a version of the slowpath that is optimized for the case where we know ahead of time we won’t be rewriting
Marius Wachtler
@undingen
Jun 29 2015 19:16
Had the same idea but never tried it out. Would we do this for all of the function which currently can rewrite them self?
Chris Toshok
@toshok
Jun 29 2015 19:17
yeah I did it for callattr, runtimecall, setattr, getattr*
and typeLookup
Marius Wachtler
@undingen
Jun 29 2015 19:17
oh :-), any perf change?
Chris Toshok
@toshok
Jun 29 2015 19:17
not really stellar, but non-zero:
                           88eedd39ce0745f6e0:             776f2ae:
       django_template.py             4.7s (2)             4.6s (2)  -2.2%
            pyxl_bench.py             3.9s (2)             3.9s (2)  -1.2%
sqlalchemy_imperative2.py             5.1s (2)             4.9s (2)  -2.3%
        django_migrate.py             1.8s (2)             1.7s (2)  -3.3%
      virtualenv_bench.py             7.8s (2)             7.7s (2)  -1.3%
                  geomean                 4.2s                 4.1s  -2.1%
Marius Wachtler
@undingen
Jun 29 2015 19:19
-2% for such a change is not bad. Would have thought it may not be noticeable at all
Chris Toshok
@toshok
Jun 29 2015 19:28
there are a few other slowpaths that I haven’t done (delattr’s one)
Travis Hance
@tjhance
Jun 29 2015 19:28
cool
Chris Toshok
@toshok
Jun 29 2015 19:28
executable is only 12k larger. i would have expected more
Kevin Modzelewski
@kmod
Jun 29 2015 19:52
oh nice
2% for that seems pretty good
Chris Toshok
@toshok
Jun 29 2015 19:53
i found I hadn’t template’d some of the things under the call path (callCLFunc, rearrangeArguments, etc)
Kevin Modzelewski
@kmod
Jun 29 2015 19:57
@rudi-c that's a transient error that only happens on travis-ci
I'm trying to look into it, but it's slow since it's not easily reproducible
Rudi Chen
@rudi-c
Jun 29 2015 19:57
ok cool, just checking
Kevin Modzelewski
@kmod
Jun 29 2015 19:57
for now, just have travis-ci rerun the build and it should go away
Chris Toshok
@toshok
Jun 29 2015 22:43
i need to stop using LD_PRELOAD of jemalloc. or i need to put it in my .profile
i keep being bitten by what I think is a perf regression but it’s just that I didn’t export LD_PRELOAD before running investigate.py
Daniel Agar
@dagar
Jun 29 2015 23:06
I'm still planning to pull jemalloc into the build system
I'm travelling
Chris Toshok
@toshok
Jun 29 2015 23:08
ah cool. was actually going to ask how the PR was going
Chris Toshok
@toshok
Jun 29 2015 23:31
fatal error: error in backend: IO failure on output stream.
errors i’ve never seen from clang #45