These are chat archives for dropbox/pyston

27th
Apr 2015
Kevin Modzelewski
@kmod
Apr 27 2015 01:05
it might just be a matter of adding an __eq__ to instancecls that defers to the class-object's `_eq`?
I think there are some similar cases in classobj.cpp
Travis Hance
@tjhance
Apr 27 2015 04:24
hm does anybody have any idea where the pointer 0x2e2e2e2e2e2e2e2e could come from?
like, if a pointer variable somewhere has a garbage value, and it’s that
what might the source be
ohhh
“.” * 10000000 would do it
Travis Hance
@tjhance
Apr 27 2015 06:05
in PystonType_GenericAlloc
we have const size_t size = _PyObject_VAR_SIZE(cls, nitems + 1);
why the +1?
Also BoxedTuple::operator new has nelts + 1, for presumably the same reason
BoxedString::operator new also has a +1
oh is it just a sentinel
there’s a comment saying it’s just a sentinel
interesting that we’re just blanket-assuming there’s a sentinel for every object?
like shouldn’t we decide whether we want a sentinel on a class-by-class basis?
why do we want a sentinel for either string or tuple?
Travis Hance
@tjhance
Apr 27 2015 06:10
also I think the +1 is double-applied
in the second overload of BoxedString::operator new
Travis Hance
@tjhance
Apr 27 2015 06:50
ummmmm
it seems to be possible for the garbage collector to be recursively run
#0  0x00000000008f15ad in pyston::gc::SmallArena::Bitmap<1024>::isSet (this=0x12758c4018, idx=188) at src/gc/heap.h:221
#1  0x00000000008f16f7 in pyston::gc::SmallArena::allocationFrom (this=0x2032950 <pyston::gc::global_heap>, ptr=0x12758c4bc8) at src/gc/heap.cpp:290
#2  0x00000000008ec096 in pyston::gc::Heap::getAllocationFromInteriorPointer (this=0x2032950 <pyston::gc::global_heap>, ptr=0x12758c4bc8) at src/gc/heap.h:525
#3  0x00000000008ec88b in pyston::gc::GCVisitor::visitPotential (this=0x7fffffff98f0, p=0x12758c4bc8) at src/gc/collector.cpp:225
#4  0x00000000008ec9e1 in pyston::gc::GCVisitor::visitPotentialRange (this=0x7fffffff98f0, start=0x12758c4bf0, end=0x12758c4c00) at src/gc/collector.cpp:239
#5  0x00000000008ece9b in pyston::gc::markPhase () at src/gc/collector.cpp:291
#6  0x00000000008ed73c in pyston::gc::runCollection () at src/gc/collector.cpp:355
#7  0x00000000008f04e4 in pyston::gc::registerGCManagedBytes (bytes=32) at src/gc/heap.cpp:119
#8  0x00000000006858c1 in pyston::gc::SmallArena::alloc (this=0x2032950 <pyston::gc::global_heap>, bytes=32) at src/gc/heap.h:179
#9  0x0000000000685892 in pyston::gc::Heap::alloc (this=0x2032950 <pyston::gc::global_heap>, bytes=32) at src/gc/heap.h:496
#10 0x00000000006855c6 in gc_alloc (bytes=24, kind_id=pyston::gc::CONSERVATIVE) at src/gc/gc_alloc.h:46
#11 0x00000000008f627c in pyston::StlCompatAllocator<std::_List_node<pyston::Box*> >::allocate (this=0x7fffffffa068, n=1) at src/gc/heap.h:53
#12 0x00000000008f61fc in std::_List_base<pyston::Box*, pyston::StlCompatAllocator<pyston::Box*> >::_M_get_node (this=0x7fffffffa068) at /home/tjhance/pyston_deps/gcc-4.8.2-install/lib/gcc/x86_64-unknown-linux-gnu/4.8.2/../../../../include/c++/4.8.2/bits/stl_list.h:334
#13 0x00000000008f6130 in std::list<pyston::Box*, pyston::StlCompatAllocator<pyston::Box*> >::_M_create_node<pyston::Box* const&> (this=0x7fffffffa068, __args=@0x7fffffff9df8: 0x327098a020)
    at /home/tjhance/pyston_deps/gcc-4.8.2-install/lib/gcc/x86_64-unknown-linux-gnu/4.8.2/../../../../include/c++/4.8.2/bits/stl_list.h:502
#14 0x00000000008f60f4 in std::list<pyston::Box*, pyston::StlCompatAllocator<pyston::Box*> >::_M_insert<pyston::Box* const&> (this=0x7fffffffa068, __position=..., __args=@0x7fffffff9df8: 0x327098a020)
    at /home/tjhance/pyston_deps/gcc-4.8.2-install/lib/gcc/x86_64-unknown-linux-gnu/4.8.2/../../../../include/c++/4.8.2/bits/stl_list.h:1561
#15 0x00000000008f09d5 in std::list<pyston::Box*, pyston::StlCompatAllocator<pyston::Box*> >::push_back (this=0x7fffffffa068, __x=@0x7fffffff9df8: 0x327098a020)
    at /home/tjhance/pyston_deps/gcc-4.8.2-install/lib/gcc/x86_64-unknown-linux-gnu/4.8.2/../../../../include/c++/4.8.2/bits/stl_list.h:1016
#16 0x00000000008f08d6 in pyston::gc::_doFree (al=0x327098a018, weakly_referenced=0x7fffffffa068) at src/gc/heap.cpp:154
#17 0x00000000008f3f94 in pyston::gc::sweepList<pyston::gc::HugeArena::HugeObj, pyston::gc::HugeArena::freeUnmarked(std::list<pyston::Box*, pyston::StlCompatAllocator<pyston::Box*> >&)::$_4>(pyston::gc::HugeArena::HugeObj*, std::list<pyston::Box*, pyston::StlCompatAllocator<pyston::Box*> >&, pyston::gc::HugeArena::freeUnmarked(std::list<pyston::Box*, pyston::StlCompatAllocator<pyston::Box*> >&)::$_4) (head=0x3271314000, weakly_referenced=..., free_func=...) at src/gc/heap.cpp:87
#18 0x00000000008f3f0c in pyston::gc::HugeArena::freeUnmarked (this=0x2032c40 <pyston::gc::global_heap+752>, weakly_referenced=...) at src/gc/heap.cpp:774
#19 0x00000000008eedfb in pyston::gc::Heap::freeUnmarked (this=0x2032950 <pyston::gc::global_heap>, weakly_referenced=...) at src/gc/heap.h:535
#20 0x00000000008edb2f in pyston::gc::sweepPhase (weakly_referenced=...) at src/gc/collector.cpp:329
#21 0x00000000008ed758 in pyston::gc::runCollection () at src/gc/collector.cpp:357
that’s not intentional, right?
Kevin Modzelewski
@kmod
Apr 27 2015 07:37
hmm seems odd that we allocate those sentinels; does cpython do that?
Travis Hance
@tjhance
Apr 27 2015 07:38
I don’t think so
Kevin Modzelewski
@kmod
Apr 27 2015 07:38
err, at least for the non-BoxedString cases
I think we have to make sure there's a null-byte at the end since we haven't been very careful about that
Travis Hance
@tjhance
Apr 27 2015 07:38
okay for strings I could believe we need it for something
for tuples it seems really odd
Kevin Modzelewski
@kmod
Apr 27 2015 07:39
yeah, odd
Travis Hance
@tjhance
Apr 27 2015 07:39
it also just seems odd that we would add a sentinel item for all classes, especially since the ‘item’ isn’t of any particular type
Kevin Modzelewski
@kmod
Apr 27 2015 07:39
I guess we should just ask chris later
re the gc, that seems unnecessary to use a gc-allocated structure for tracking the weak references, but again we should ask chris about it
Chris Toshok
@toshok
Apr 27 2015 17:17
oh, yeah that looks very wrong
wrong that we’re reentrant, but for weakrefs we might have to call back into python code
Michael Arntzenius
@rntz
Apr 27 2015 18:03
hm, tester.py checks for TESTNAME.expected before TESTNAME.expected_cache - any reason why? is this so we can manually override expected output or something?
i wish gitter didn’t inline gists :/
Michael Arntzenius
@rntz
Apr 27 2015 18:29
hm, if LANG is set in the environment then sys_test.py produces a different line for sys.getfilesystemencoding() between us & Python
eg. LANG=C python test/tests/sys_test.py produces ANSI_X3.4-1968.
Chris Toshok
@toshok
Apr 27 2015 18:30
do we just always return UTF-8?
Michael Arntzenius
@rntz
Apr 27 2015 18:30
looks like
Michael Arntzenius
@rntz
Apr 27 2015 18:50
ruh-roh. I'm getting an assertion failure in io_test on master in the unwinder (not my unwinder, master's unwinder) - regs_valid & (1 << dwarf_num) is failing
only with -O, though, and Travis CI doesn't seem to show this problem
Michael Arntzenius
@rntz
Apr 27 2015 19:15
ok, io_test seqiter_protocol gzip_test are all failing that assert on -O, and moreover distutils_test and cpython/test_strftime are both causing LLVM to run out of registers
Michael Arntzenius
@rntz
Apr 27 2015 20:30
okay, this is really weird. exec ';' is producing a SyntaxError when compiled with cmake (against libgcc 4.9) but an assertion trips in parse_file when compiled with make (against libgcc 4.8)
Kevin Modzelewski
@kmod
Apr 27 2015 21:03
I think that last one is due to a recent change in libpypa
and make and cmake use different pypa installs
Michael Arntzenius
@rntz
Apr 27 2015 21:04
aha
Kevin Modzelewski
@kmod
Apr 27 2015 21:04
(cmake gets it from git submodules so is up to date, but your pyston_deps install might be out of date)
and yeah, the .expected files let us manually specify what we want the tests to do
either due to perf reasons (less important now) or because we have different behavior from cpython
Kevin Modzelewski
@kmod
Apr 27 2015 21:30
@undingen forgot to respond about the GMP thing -- I think it's fine if we need a patch for pycrypto, especially if it's for an optional part of the library
if you send me a patch file I can incorporate it into our internal installer
separately, I wonder what the best long-term approach is for things that want to look at the internals of objects like that
we could potentially have a convertLongToCPythonsFormat(), and we could ideally determine when it's needed
in this case it'd be silly to convert to cpython's format just to convert it back to gmp, but at least it would give us a more mechanical way of dealing with these issues
Chris Toshok
@toshok
Apr 27 2015 22:19
hm, this trying to time the slowpath is really hairy
transitioning out of the slowpath can end multiple timers
Kevin Modzelewski
@kmod
Apr 27 2015 22:20
when do we have multiple slowpaths at once?
maybe we could try to make sure that we can only be contributing to one slowpath timer at a time?
Chris Toshok
@toshok
Apr 27 2015 22:22
yeah, i think it’s just an artifact of me adding timers at places other than the toplevel extern “C” … entry functions
Chris Toshok
@toshok
Apr 27 2015 22:27
does every runtimeCall{Internal} chain terminate with the function_cls / builtin_function_or_method_cls case?
the one where we call through the CLFunction::InternalCallable
Kevin Modzelewski
@kmod
Apr 27 2015 22:29
I think so
Chris Toshok
@toshok
Apr 27 2015 22:31

this looks potentially useful then:

zzz_slowpath_callattr_us: 25350
zzz_slowpath_getattr_us: 80359
zzz_slowpath_getitem_us: 705
zzz_slowpath_nonzero_us: 7627
zzz_slowpath_setitem_us: 1738

(zzz added so they show at the end of the list). that’s output from running unicode_test in the interpreter