These are chat archives for dropbox/pyston

5th
Jun 2015
Chris Toshok
@toshok
Jun 05 2015 00:31
i’m trying to get this super() __getattribute__ change PR-ready, and am having trouble writing a test that fails with typeLookup but not with getattrInternalGeneric
without my test that abort()s in superGetAttribute. but with my change it raises AttributeError: 'super' object has no attribute ‘x'
cpython does the same thing, so...
Kevin Modzelewski
@kmod
Jun 05 2015 00:46
hmm you could probably do it with a subclass of super
hopefully not very realistic code, but that would at least let you get instance attributes on the super object
Chris Toshok
@toshok
Jun 05 2015 16:06
that gets me further. also found a cpython test for super which shows some weird behavior in cpython
Chris Toshok
@toshok
Jun 05 2015 16:26
oh, I understand now. super objects themselves act as descriptors, and can be partially bound
so something like this works:
class C(A):
    def __init__(self):
        self.__super.__init__()

C._C__super = super(C)
Travis Hance
@tjhance
Jun 05 2015 16:31
my head hurts
Marius Wachtler
@undingen
Jun 05 2015 16:37
am I supposed to understand what this does? :-D
Chris Toshok
@toshok
Jun 05 2015 16:37
haha
the __super turns (at __init__ time) into effectively super(C, self)
the C is supplied by that assignment at the bottom, partially binding the super descriptor
a use case necessitating this feature eludes me :)
Marius Wachtler
@undingen
Jun 05 2015 16:42
Is the __super a special attribute? or why does it know that this should call super()???
Chris Toshok
@toshok
Jun 05 2015 16:43
because the value of __super is a super object, and super has a tp_descr_get in cpython
tp_descr_get checks if the super object is fully bound, and if it isn’t, it creates and returns a new super object that is fully bound (equivalent to super(C, self)
it doesn’t impact the way super is normally used, like super(C, self).__init__(). only when super objects are partially bound and stored as attributes
Marius Wachtler
@undingen
Jun 05 2015 16:45
ah so the __supergets set inside A?
Chris Toshok
@toshok
Jun 05 2015 16:46
no, that’s what the _C__super assignment sets
__ attributes get mangled to include the class name
Marius Wachtler
@undingen
Jun 05 2015 16:46
oh thanks now I understand it :-)
was about time to read about the python name mangling :-D
Rudi Chen
@rudi-c
Jun 05 2015 17:45
(gdb) p *start
$3 = (void * const) 0x127014e498
(gdb) p start
$4 = (void * const *) 0x7fffffffa920
(gdb) p/x *0x7fffffffa920
$5 = 0x7014e498
Why does printing *start and *0x7fffffffa920 (the value of start) give different values? Does the leading 0x12 digits of `start related to paging or something like that?
Chris Toshok
@toshok
Jun 05 2015 17:48
Try print *(void**)0x7fff...?
Rudi Chen
@rudi-c
Jun 05 2015 17:52
Yeah that gives 0x127014e498
Marius Wachtler
@undingen
Jun 05 2015 17:56
Here is a table of gc address starts:
constexpr uintptr_t SMALL_ARENA_START = 0x1270000000L;
constexpr uintptr_t LARGE_ARENA_START = 0x2270000000L;
constexpr uintptr_t HUGE_ARENA_START = 0x3270000000L;
and for generator stacks: 0x4270000000L;
mmh didn't know the ranges of the areas except the small and generator one. Don't think I encountered the others often.
Rudi Chen
@rudi-c
Jun 05 2015 18:00
I'm curious as to how gdb knows that if the value is a pointer, it should have the 0x127 of the arena start.
Chris Toshok
@toshok
Jun 05 2015 18:01
It was just giving you the lower 32 bits I bet
Rudi Chen
@rudi-c
Jun 05 2015 18:02
Oh right, 0x7014e498 is 32 bits.
I thought it had something to do with page numbers ^^'
Chris Toshok
@toshok
Jun 05 2015 18:20
Yeah visual studio does the same thing. I don't understand why the default isn't 64bits when the platform is
Marius Wachtler
@undingen
Jun 05 2015 18:23
we can be lucky they don't use 16bit as default :-D
Chris Toshok
@toshok
Jun 05 2015 21:07
heh good point
i wonder if lldb does the right thing
Marius Wachtler
@undingen
Jun 05 2015 21:12
I tried it recently for the first time
because gdb complained about my print statement (used c++ syntax and had function calls + casts) and I couldn't figure out how it's supposed to be written. and lldb liked the same print statement :-)
Bob Fang
@dorafmon
Jun 05 2015 21:26
hi sorry to bother but make quick_check failed for me for no reasons
here is the output:
Marius Wachtler
@undingen
Jun 05 2015 21:31
for the decorator_whitespace.py error: git submodule update
Chris Toshok
@toshok
Jun 05 2015 21:31
which test failed?
Marius Wachtler
@undingen
Jun 05 2015 21:32
you are running an older libpypa (=parser) (not much older good just changed a few mins ago :-D)
and for the weakref4.py see dropbox/pyston#587
Bob Fang
@dorafmon
Jun 05 2015 21:33
so this means everybody is getting an failed quick check right now?
if #587 is not solved
Chris Toshok
@toshok
Jun 05 2015 21:35
as people update to master, yeah. working on side feature branches, though, has kept some from being affected
Marius Wachtler
@undingen
Jun 05 2015 21:39
The issue surfaced today (/yesterday) so it's not like we are all ignoring that one since ages. It properly depends on specific stack layout. That's why some hit it and some not.
@kmod what the cause of an issue like the travis-ci gcc error for you generator change?
/home/travis/build/dropbox/pyston/src/codegen/parser.cpp:1212: pyston::AST_Module* pyston::caching_parse_file(const char*): Assertion `tries <= 5' failed: repeatedly failing to parse file
Chris Toshok
@toshok
Jun 05 2015 21:41
yeah, I ran into it a few days ago but was sure it was a problem on my branch, and finally committed the workaround: toshok/pyston@3307cef
@rudi-c had an interesting insight in that maybe it wasn’t the recursive function possibly clearing the stack (since my “fix” just calls the same function, which should clear the same stack, twice) - that maybe it was just the fact that there was another statement at all
Kevin Modzelewski
@kmod
Jun 05 2015 21:44
@undingen dunno, usually I just ask travis-ci to rerun it (just did for those)
haven't felt like doing another set of parsing investigations :P
Rudi Chen
@rudi-c
Jun 05 2015 21:45
@toshok So I just found another way pointers get retained.
There's the register r14 pointing to some random heap object (xrange). This register gets reused at some point (some instruction sets the lower 8 bits to 1). This register (the full contents of which are garbage, but the lower 8 bits are used for some reason) gets pushed on the stack by function calling conventions.
Now, there's a misaligned heap pointer on the stack, but we handle interior pointers.
The pointer corresponding to this valid interior pointer happens to be an object I want to clear.
Basically in this case, it's not even a problem of having a local variable that doesn't get cleared from the stack. It's a register containing garbage value (but is the interior pointer of a valid heap address) that happens to be on the stack due to callee-save.
This totally goes into my collection of obscure edge cases encountered :D
Kevin Modzelewski
@kmod
Jun 05 2015 21:53
ok hopefully #590 should make weakref4 non-flaky once and for all :P
Chris Toshok
@toshok
Jun 05 2015 21:56
@rudi-c yeah that’s the bet with conservatively scanning stacks - that pointers like that are rare. is there any way to tell what frame it corresponds to?
btw, i tried switching the gc to only deal with actual pointers to the objects (not interior pointers), and nothing crashed
Rudi Chen
@rudi-c
Jun 05 2015 21:58
Yeah, it's in callattrinternal or something because one of it's callers sets the lower 8 bits of r14.
Chris Toshok
@toshok
Jun 05 2015 21:58
i wonder if that wouldn’t be something to try somehow… a command line switch, and adding that configuration to the tester matrix
Bob Fang
@dorafmon
Jun 05 2015 21:58
one quick question, did we write the GC ourselves or we reused some GC from other project?
Chris Toshok
@toshok
Jun 05 2015 21:58
it’s not reused
all homegrown :)
Rudi Chen
@rudi-c
Jun 05 2015 21:59
@kmod mentioned we need interior pointers in cases like interating through a string character by character and the compiler optimizing away the original pointer
Chris Toshok
@toshok
Jun 05 2015 21:59
yeah, for that we’ll need to start using the Rooted<> classes
Rudi Chen
@rudi-c
Jun 05 2015 22:00
?
Chris Toshok
@toshok
Jun 05 2015 22:00
err, StackRoot<> classes
Rudi Chen
@rudi-c
Jun 05 2015 22:00
What are they for again?
Chris Toshok
@toshok
Jun 05 2015 22:01
you’d do { RootedBoxedString r(string); /* do something where you iterate over string’s characters */ }
Rudi Chen
@rudi-c
Jun 05 2015 22:01
Ah I see.
It seems like it'd be really easy to forget to use them though, and it would crash once in a blue moon.
Marius Wachtler
@undingen
Jun 05 2015 22:05
Have you measured any speedup with disabled interior pointers?
noticable speedup
Bob Fang
@dorafmon
Jun 05 2015 22:07
#590 failed...
Rudi Chen
@rudi-c
Jun 05 2015 22:08
That's weird, it has nothing to do with weakref4
Chris Toshok
@toshok
Jun 05 2015 22:10
@undingen I didn’t check. would be interesting to see though
@rudi-c yeah that’s why having tester (maybe only on travis) do another run with that config might be a good idea. i didn’t notice any crashes here, even running django-template, but other longer running integration tests might have issues
Chris Toshok
@toshok
Jun 05 2015 23:11
do we have any known issues re: descriptors?