These are chat archives for dropbox/pyston

21st
May 2015
Rudi Chen
@rudi-c
May 21 2015 00:06
In our GC we collect weak reference that can be freed, deallocate them in one pass, then call the callbacks in another pass. In theory it should be possible for the callback to 'resurrect' other weak references. I guess this is not an edge case worth handling right? Otherwise we'd need a callback ordering algorithm too.
Kevin Modzelewski
@kmod
May 21 2015 01:18
I don't think that particular case is possible
that second pass calls the callbacks on live weakrefs that point to just-collected objects
so if they have a way of referencing an object, that object is still live
oh, I guess they could look at other weak references and expect them to not have been cleared yet
there was a thread about that on the pypy list a couple months ago
err, I think that thread might have been about an object getting resurrected but its weakrefs were already cleared
I guess my thought is that we will definitely start running into weird ordering issues, but it might be hard to predict which ones we'll hit
Travis Hance
@tjhance
May 21 2015 02:45
how much time have we spent optimizing the assembly that the rewriter outputs?
have we already spent a lot of work there?
or not
Kevin Modzelewski
@kmod
May 21 2015 02:54
I think the last work we did on that was what you did with the better register allocation :)
Travis Hance
@tjhance
May 21 2015 02:56
that was a while ago
maybe i’ll try to look into this sometime soon
Kevin Modzelewski
@kmod
May 21 2015 02:56
marius had that pr for improving our loading of constants
Travis Hance
@tjhance
May 21 2015 02:56
oh yeah
i gave feedback on it
Kevin Modzelewski
@kmod
May 21 2015 03:13
oh man, we can now start metaserver without crashing :)
(counting that as a goal we hit this past week :P )
Travis Hance
@tjhance
May 21 2015 03:13
!!!!!!!!!
Kevin Modzelewski
@kmod
May 21 2015 03:14
I have to comment out all the places that we import ncrypt/numpy/openssl
Travis Hance
@tjhance
May 21 2015 03:14
oh
cheaterrr
Kevin Modzelewski
@kmod
May 21 2015 03:14
and it's with a build of Pyston that doesn't pass the tests :(
but hey, some nice progress in the past three months since the 0.3 release, when we could barely run distutils :)
Marius Wachtler
@undingen
May 21 2015 08:08
:+1:
Marius Wachtler
@undingen
May 21 2015 08:15
@tjhance the PR is still up dropbox/pyston#310 but I forgot about it and didn't address the comments.
Rudi Chen
@rudi-c
May 21 2015 22:59
For tp_dealloc (currently ignored) and simple_destructor (defined for a few types), I'm thinking we could replace simple_destructor with a boolean like "destructor_is_safe_to_call_at_any_time" which would default to false and be set to true for optimizing objects we know are safe to call the destructor (tp_dealloc or user-defined) on without going through the destructor ordering constraints (like files).
Rudi Chen
@rudi-c
May 21 2015 23:06
Also, anybody else has problems with virtualenv timing out? It's taking 3-4 minutes for me (my GC modifications only add ~10 seconds).
Travis Hance
@tjhance
May 21 2015 23:07
yeah actually the other day make check wouldn't finish, I'm not sure which test case it was stuck on
but it was probably that one
Kevin Modzelewski
@kmod
May 21 2015 23:09
yeah the virtualenv test takes ~5 minutes now
in debug mode
I usually just do make quick_check before submitting the pr and let travis-ci do the waiting :P
Rudi Chen
@rudi-c
May 21 2015 23:10
Ohh neat didn't know about that one
Chris Toshok
@toshok
May 21 2015 23:19
so i’m trying to get jitdev to the point where it can build the hacked up libgcc again, but I think ubuntu may have made that impossible
installing gcc-4.8-multilib installs gcc-4.9
actually their gcc/g++ dependencies are pretty screwy in general. uninstalling gcc-5-base would uninstall both gcc-4.9 and gcc-4.8
Michael Arntzenius
@rntz
May 21 2015 23:27
@kmod Do you know where in the build system is responsible for applying the libunwind patches? I'm looking at CMakeLists.txt and it doesn't seem to have anything relevant in there...
Kevin Modzelewski
@kmod
May 21 2015 23:28
hmm maybe there isn't
Michael Arntzenius
@rntz
May 21 2015 23:34
looks like specifying a PATCH_COMMAND might work
Rudi Chen
@rudi-c
May 21 2015 23:37

Uh interesting. I can comment out

    // Zero out the CPython tp_* slots:
    memset(&tp_name, 0, (char*)(&tp_version_tag + 1) - (char*)(&tp_name));

from the BoxedClass constructor and none of the basic tests fail.

Kevin Modzelewski
@kmod
May 21 2015 23:39
oh, yeah I think the allocator zeros that memory now
oh nevermind, I thought it did, but I guess it doesn't for classes
or things that go through the "SIMPLE" allocators
Rudi Chen
@rudi-c
May 21 2015 23:42
Is it necessary to zero out the tp_* slots, or is that just a temporary thing we do for now to get things running?
Kevin Modzelewski
@kmod
May 21 2015 23:42
I don't think we write to all of them yet
maybe once we do, we could remove the memset
(ie once we implement all the slotdefs in typeobject.cpp)
Michael Arntzenius
@rntz
May 21 2015 23:43
@kmod hm, the one patch already in the libunwind_patches doesn't seem to apply (it changes include/libunwind.h rather than include/libunwind.h.in)
perhaps it's meant to be applied post-configure, but CMake appears to want to apply patches pre-configure
Kevin Modzelewski
@kmod
May 21 2015 23:44
yeah I think we used to apply that against the installed version of libunwind, not the source
we don't need that one anymore, it got mainlined
Michael Arntzenius
@rntz
May 21 2015 23:47
ok, I'm just using it to test out how PATCH_COMMAND works
Rudi Chen
@rudi-c
May 21 2015 23:48
So the aim is to reimplement all the slots ourselves? What about C extensions?
Kevin Modzelewski
@kmod
May 21 2015 23:49
well, the slotdefs say how the c slots correspond to python attributes
the goal is to import all of them
we just haven't done it yet
I think at this point it should be pretty easy to get the rest in
Rudi Chen
@rudi-c
May 21 2015 23:55
For tpdealloc, it's possible to implement one in C without having a corresponding del function (however, if _del is defined, it gets called from as/from tp_dealloc).
Ah markdown gobbles my underscores...
Chris Toshok
@toshok
May 21 2015 23:56
right. cpython wraps certain __methods__ and exposes them via the tp_slots
tp_* slots, that is. isn’t there already tp_slots?
Rudi Chen
@rudi-c
May 21 2015 23:57
There's no tp_slots
Anyway, I'm just wondering if it's worth supporting the case where tp_dealloc is defined (in C code, presumably as an extension) but there's no __del__ method.
(That case does exist...right...?)