These are chat archives for dropbox/pyston

20th
May 2015
Chris Toshok
@toshok
May 20 2015 00:00
descr_test.pyston.so looks to have the same issue
k, will add the deps to multiprocessing
Chris Toshok
@toshok
May 20 2015 00:44
oof, that switch from small heap to large heap for BoxedClass comes at a pretty large cost
iter_gc.py takes 1.7s if the largest size allocated from the small arena is 1024 bytes. The shape change bulks them up to 1048 bytes
if I change the largest bucket size to 1280, iter_gc.py takes 0.9 seconds
Chris Toshok
@toshok
May 20 2015 05:57
does the non-cmake pyston_release work for anyone?
Kevin Modzelewski
@kmod
May 20 2015 07:25
I would definitely encourage you to speak with @dagar if you're running into any issues between the cmake / makefile build
if we can sort those out, the cmake build system definitely offers some nice advantages :)
Chris Toshok
@toshok
May 20 2015 07:32
oddly enough it was something that should have broken the cmake build too. not sure why it wasn't
I had a static c++ property whose initializer did a gc alloc
Kevin Modzelewski
@kmod
May 20 2015 07:40
yeah who knows, maybe the conservative GC could be picking it up in one build but not the other
I wish we could have it fail more predictably
also, I hope my comments on your pr aren't too out of date :)
Chris Toshok
@toshok
May 20 2015 07:42
they aren’t - and actually I was at this moment addressing one of them (invalidation :)
turns out that without invalidation, the shape is only assigned on the first typeLookup, which is too late to reuse an existing ic slot
i was hoping to avoid needing invalidation to handle our startup and lexically defined classes (and only need it for actual class manipulation)
Kevin Modzelewski
@kmod
May 20 2015 07:46
do we call typeLookup during typeNew at all?
Chris Toshok
@toshok
May 20 2015 07:47
i don’t think so
Chris Toshok
@toshok
May 20 2015 08:24
adding the assert for BoxedClass inside computeTotalShape crashes things pretty quick. will need to figure out a way to address old style classes
but for now, sleep
Chris Toshok
@toshok
May 20 2015 15:31
hm, what if we just used the HiddenClass at each step of the mro to determine total shape
instead of hashing local attributes
Rudi Chen
@rudi-c
May 20 2015 22:00
import weakref
import gc

def callback(wr):
    print "object was destroyed", wr()

def retainer(ref):
    def cb(wr):
        print "object was destroyed", ref, wr()
    return cb

class C(object):
    def __init__(self, index):
        self.index = index
    def __del__(self):
        print "deleted", self.index

def test():
    c1 = C(1)
    c2 = C(2)
    x = (weakref.ref(c1, callback), weakref.ref(c2, retainer(c2)))

wr = test()
gc.collect()
gc.collect()
gc.collect()
output (with CPython):
object was destroyed None
deleted 1
deleted 2
Any idea why the callback for c2 doesn't get called but the finalizer does?
Chris Toshok
@toshok
May 20 2015 22:01
my guess is the weakref is collected at the same time as c2?
maybe try adding the weakrefs to a module-level global so that they’re guaranteed to stay alive
Rudi Chen
@rudi-c
May 20 2015 22:02
I'm trying to see what happens when weakref callbacks and finalizers are mixed together
and when the object dies
Chris Toshok
@toshok
May 20 2015 22:04
oh, the method returns a tuple of the weakrefs
so they should be alive, right
actually that’s interesting that c2 is deleted at all
Rudi Chen
@rudi-c
May 20 2015 22:05
c1 should and is deleted for sure
Chris Toshok
@toshok
May 20 2015 22:06
there’s a closure created for cb that should create a strong reference to ref (c2)
Rudi Chen
@rudi-c
May 20 2015 22:06
I feel like c2 should be deleted after the callback?
But the callback doesn't get called
Chris Toshok
@toshok
May 20 2015 22:06
and the weakref callbacks are referenced strongly from the weakrefs themselves
so essentially you have a ref chain: weakref -> callback -> closure -> c2
are the deleted lines printed when one of the gc.collects is called? maybe put print’s between them and after the last one to see which one causes them to be marked as garbage
maybe we’re deleting 2 when the module goes away (if we do a final GC, not sure if we do)
Rudi Chen
@rudi-c
May 20 2015 22:16
before the GC calls
Theory: In the case of c1, there's only a weak ref. When the weakref goes away, the object gets destroyed, so both the callback and destructor are called. In the case of c2, there is both a weakref and a strongref from the callback. When the weakref goes away, the callback is not called because c2 still has a strongref count. Then the callback goes away. Then the strongref count goes to 0 and the destructor is called. But there's already no more weakref or callback by that time.
Chris Toshok
@toshok
May 20 2015 22:20
oh actually x isn’t returned from test()
Rudi Chen
@rudi-c
May 20 2015 22:20
yeah it used to be but I changed and still have wr = test() ^^'
Chris Toshok
@toshok
May 20 2015 22:20
okay, so the weakrefs aren’t retained either
Rudi Chen
@rudi-c
May 20 2015 22:21
If I return it, same things get printed except that "deleted 2" is printed after the GC calls instead of before.
Chris Toshok
@toshok
May 20 2015 22:22
i’m confused - so the output comes before any of the gc.collect() calls happen?
Rudi Chen
@rudi-c
May 20 2015 22:22
Without returning :
object was destroyed None
deleted 1
deleted 2
A
B
C
With returning :
object was destroyed None
deleted 1
A
B
C
deleted 2
Where ABC are GC calls
Chris Toshok
@toshok
May 20 2015 22:23
gah: "output (with CPython):"
i missed that
okay this is all explainable then :)
in cpython I believe the gc.collect() does nothing but invoke the cycle collector
all destruction is done at the time things leave scope or are overwritten
Rudi Chen
@rudi-c
May 20 2015 22:24
Looks like it yup. I'll keep it as a nice "what happens?" puzzle for Python users.
Chris Toshok
@toshok
May 20 2015 22:29
so yeah, the reason the callback isn’t called for the second weakref is by the time c2 is deleted the weakref is already gone (since it held a strong ref to c2)
interesting, and I’m not sure I’d get that right on first viewing :)