These are chat archives for dropbox/pyston

28th
Jul 2015
Kevin Modzelewski
@kmod
Jul 28 2015 01:23
hey guys, I just pushed some changes to investigate.py that should cut down on the run-to-run variance
which was pretty high for django_template3
I'd recommend pulling the new changes -- you should wipe your saved_runs directory though
Sun
@Daetalus
Jul 28 2015 06:49
Hi, I am trying to fix test_int. int() function have some problems. For old stlyle class, I solved by add a intanceIntinto classobject.cpp. But for new style class. According to the gdb trace, it will not call PyNumber_Int, instead call intNew directly. Which will get wrong error message.
I will investigate it more, but if someone could give some tips or tell me that I am in wrong direction, it will be very helpful.
Kevin Modzelewski
@kmod
Jul 28 2015 07:19
not sure I 100% follow, but it looks like our int() stuff is pretty different from CPython's
if it's just an error-message thing I think it's fine to not worry about that right now
if there are behavioral differences we might want to switch to the cpython code
Dong-hee Na
@corona10
Jul 28 2015 07:29
can you reveiw #766
?
Sun
@Daetalus
Jul 28 2015 07:30

For example:

class Foo(object):
    pass

bar = Foo()
int(bar)

In here, int() should call PyNumber_Int in abstract.cpp, or call intNew in int.cpp?

Kevin Modzelewski
@kmod
Jul 28 2015 07:31
well, I guess I would say that int() is intNew
but probably intNew should call PyNumber_Int
it corresponds to cpython's int_new, which does call PyNumber_Int
Sun
@Daetalus
Jul 28 2015 07:32
Ok, thanks, I will investigate more!
Another question, I am trying to get fully support for old style class. Does rcmp need to be implemented? According CPython documentation, rcmp no longer supported since 2.1
Kevin Modzelewski
@kmod
Jul 28 2015 07:35
what does cpython do? :)
Travis Hance
@tjhance
Jul 28 2015 07:35
I can’t even find anything about rcmp on google
Kevin Modzelewski
@kmod
Jul 28 2015 07:36
it looks like there aren't any references to __rcmp__ in the cpython codebase
Sun
@Daetalus
Jul 28 2015 07:36
I will experiment it in CPython, never used it before.
Travis Hance
@tjhance
Jul 28 2015 07:36
the top hits for “python rcmp” are this unrelated python package https://pypi.python.org/pypi/rcmp/0.7 and the 2013 New Brunswick python attack https://en.wikipedia.org/wiki/2013_New_Brunswick_python_attack
(which involves the Royal Canadian Mounted Police, i.e., RCMP)
Kevin Modzelewski
@kmod
Jul 28 2015 07:37
haha
Travis Hance
@tjhance
Jul 28 2015 07:37
oh @kmod I’m working on the slowpath thing
it seems straightforward, I think
well, we’ll se
e
Kevin Modzelewski
@kmod
Jul 28 2015 07:39
awesome :)
Sun
@Daetalus
Jul 28 2015 07:44
CPython doesn't support rcmp now, at least for 2.7.6
Obviously.... The document already said that...
Marius Wachtler
@undingen
Jul 28 2015 08:04
I would have thought that the template CAPI/C++ API approach would force us to transfer a lot of the code into the header files. But I'm pleased by how good this integrates.
Kevin Modzelewski
@kmod
Jul 28 2015 08:23
yeah, I didn't realize that you could call templates that you don't have the source to :)
I need to figure out how to reduce the crazy boilerplate / code duplication
I'm not sure if "crazy macros" is or worse though
Marius Wachtler
@undingen
Jul 28 2015 08:30
:-(
Just saw that that running raytrace in -I makes the baselinejit use 2GB of JITed code :-P
it's surprisingly fast for that amount of work... :-D
Kevin Modzelewski
@kmod
Jul 28 2015 10:24
haha :)
Rudi Chen
@rudi-c
Jul 28 2015 20:34
NumPy uses the Ellipsis object x_x
I can get around it by adding stub functions for now I think. It's a really nice starter project for someone who wants to contribute to Pyston, it's something I'd like to leave to a contributor :)
Marius Wachtler
@undingen
Jul 28 2015 21:00
concernign the pgo: is this enabled per default? Or how do I get a pgo build + do the training + get the executable with profiling?
Marius Wachtler
@undingen
Jul 28 2015 21:16
and does this do lto?
Kevin Modzelewski
@kmod
Jul 28 2015 21:32
you can do make pyston_release_gcc_pgo (or just make pyston_pgo with the change I just pushed)
looks like there's an issue with it though where it's not playing nice with the assembly file you added recently
hmm it's not including the compiled .S in the link line
Marius Wachtler
@undingen
Jul 28 2015 21:35
oh :-(
pgo is only enabled with gcc?
Kevin Modzelewski
@kmod
Jul 28 2015 21:37
oh hmm not sure what happened but it re-ran cmake and got past that assembly-file issue
but yeah it's only for gcc right now, sounded like clang's pgo works pretty differently
Marius Wachtler
@undingen
Jul 28 2015 21:38
ok I will try it out
Chris Toshok
@toshok
Jul 28 2015 21:41
@kmod oh, build’s working now?
Kevin Modzelewski
@kmod
Jul 28 2015 21:41
well it hits an issue with my templating change
but yes got past the assembly-file thing :P
Marius Wachtler
@undingen
Jul 28 2015 21:45
just got the same error (../../src/runtime/generator.cpp:102: error: undefined reference to 'pyston::Box* pyston::callCLFunc<(pyston::ExceptionStyle::ExceptionStyle) blabla
Kevin Modzelewski
@kmod
Jul 28 2015 21:47
#773
Marius Wachtler
@undingen
Jul 28 2015 21:54
works :-)
Kevin Modzelewski
@kmod
Jul 28 2015 21:56
the pgo build times aren't that bad once you have it built the first time :)
Rudi Chen
@rudi-c
Jul 28 2015 21:57
NumPy is so big it triggers the assertion RELEASE_ASSERT(bss_end - bss_start < 100000, "Large BSS section detected - there maybe something wrong");
Marius Wachtler
@undingen
Jul 28 2015 21:59
:-(, I choose this number for no specific reason, you can increase it if you like. I just thought that if it's very large it's probably a error.
Rudi Chen
@rudi-c
Jul 28 2015 22:00
Yeah I just added a zero. I just found it funny.
Actually, NumPy has a bunch of weird .c.src files that have their own custom macro syntax and macro preprocessor.
Marius Wachtler
@undingen
Jul 28 2015 22:01
but it may make the GC visists very expensive :-(
Rudi Chen
@rudi-c
Jul 28 2015 22:01
So you see things like "@name@_Descr" and it expands into multiple copies for name in LONG, LONGLONG, etc
Poor man
's template system
Why would it make GC visits expensive?
Marius Wachtler
@undingen
Jul 28 2015 22:03
because it conservatively visits the whole bss section (the static vars inside the shared library)...
Rudi Chen
@rudi-c
Jul 28 2015 22:03
Every time?
Marius Wachtler
@undingen
Jul 28 2015 22:03
I don't really know if >(100000/8) pointers is significant
yes
I added it because it's pretty easy to miss GC registering static variables which store python objects when integrating external code
it even hit us several times during importing of cpythons builtin modules
Rudi Chen
@rudi-c
Jul 28 2015 22:06
Non-type static variables?
*Non-type static python objects?
Marius Wachtler
@undingen
Jul 28 2015 22:06
I think it's all static non const variables
so it may also be a int etc
Rudi Chen
@rudi-c
Jul 28 2015 22:07
Have you seen non-type python objects directly defined as static structs before? Or just things that reference python objects.
NumPy has a bunch of those which are going to be a pain because it's not obvious how you'd find them and register them as non-heap roots.
Marius Wachtler
@undingen
Jul 28 2015 22:09
If I understand you correctly I saw booth. Problematic is if the code directly uses malloc and stores inside the malloced mem pointers to python objects...
we will not scan them :-(
Rudi Chen
@rudi-c
Jul 28 2015 22:10
Yeah, it's probably impossible to completely guarantee that we know about all non-heap Python objects.
Marius Wachtler
@undingen
Jul 28 2015 22:10
which I think could be very common for extensions written in c++. E.g. c++ class which stores pointers to python objects and get's alloced with new...
I think with scanning the bss section we should be safe for non heap objects. But the heap allocated ones (not using our gc alloc funcs) I fear the most.
Rudi Chen
@rudi-c
Jul 28 2015 22:12
So the non-pyston-heap heap.
Marius Wachtler
@undingen
Jul 28 2015 22:14
exactly, we could hook malloc and free to catch most of them but this may slow down everything and requires us to change pystons code to use the unhooked when internally required :-(
I think I had it hooked for lxml but it was really slow :-(
Chris Toshok
@toshok
Jul 28 2015 22:15
it’ll be even slower now that we’ll be switching away from jemalloc :/
Rudi Chen
@rudi-c
Jul 28 2015 22:15
We are?
Chris Toshok
@toshok
Jul 28 2015 22:15
oh, we wouldn’t replace malloc, we’d just register the returned memory regions as places to scan?
for some reason I was assuming we’d just replace malloc with our own that allocated from the gc heap (conservatively)
Marius Wachtler
@undingen
Jul 28 2015 22:16
I had it allocate the mem with out GC instead of malloc. but just registering the bytes may be better
Travis Hance
@tjhance
Jul 28 2015 22:27
we’re switching away from jemalloc?
what was wrong with it?
Marius Wachtler
@undingen
Jul 28 2015 22:30
no it think chris just meant that if we would hook malloc and replace all aloccs with GC allocs we would losse the perf win of jemalloc for shared libraries
Denis Denisov
@denji
Jul 28 2015 23:16