These are chat archives for dropbox/pyston

4th
Aug 2015
Kevin Modzelewski
@kmod
Aug 04 2015 01:06
man, another reason to dislike using distutils to build our standard sharedmodules (like ctypes):
it doesn't rebuild them if we change our header files
took me a while to figure out why it was crashing
turns out that one of the shared modules didn't pick up the new attribute I added, so when you write to it, it clobbers whatever happens to be next in memory :P
Travis Hance
@tjhance
Aug 04 2015 04:12
hi
okay so given that the bjit apparently gives bad stacktraces, how can I debug my code?
Kevin Modzelewski
@kmod
Aug 04 2015 04:17
hmm what ends up failing, is it just a segfault?
Travis Hance
@tjhance
Aug 04 2015 04:17
SIGSEGV
Kevin Modzelewski
@kmod
Aug 04 2015 04:17
maybe try to guess what kind of object it's looking it and what the segfault could mean
ie is it that an object got corrupted, or maybe it got freed
or maybe it's just some totally random pointer
Travis Hance
@tjhance
Aug 04 2015 04:18
is there a way to disable the bjit?
Kevin Modzelewski
@kmod
Aug 04 2015 04:18
and then maybe just try to whittle down on the changes you made :/
Travis Hance
@tjhance
Aug 04 2015 04:18
it’s all-or-nothing
Kevin Modzelewski
@kmod
Aug 04 2015 04:18
oh, yeah you can do -n to bypass the interpreter+bjit, and I think there's a flag to disable just the bjit part
Travis Hance
@tjhance
Aug 04 2015 04:19
of course it’s possible that my changes are specifically bad when dealing with the baseline jit, since it subclasses the rewriter in a way I don’t really understand
no, it still fails with -n
hm, what’s the difference between SIGBUS and SIGSEGV?
Kevin Modzelewski
@kmod
Aug 04 2015 04:21
haha
not quite sure, maybe you hit a different type of memory address (one that is mapped but you can't write to?)
you could try the "disable your change after getting invoked N times and then binary search on N" trick
Travis Hance
@tjhance
Aug 04 2015 04:22
maybe it jumped to a bad instruction
yeah that trick is really baller
Kevin Modzelewski
@kmod
Aug 04 2015 04:22
I think that's SIGILL
Travis Hance
@tjhance
Aug 04 2015 04:22
anyway it’s already failing inside an IC
i can probably manage from here, thanks!
Sun
@Daetalus
Aug 04 2015 06:57

Hi, the from future_builtins import hex, ... still not work. Here is the error message:

Last line of stderr: ImportError: cannot import name hex
/home/sun/pyston2/test/tests/future_builtins.py:
Traceback (most recent call last):
  File "./tools/tester.py", line 444, in worker_thread
    results[job[0]] = run_test(*job)
  File "./tools/tester.py", line 164, in run_test
    return determine_test_result(fn, opts, code, out, stderr, elapsed)
  File "./tools/tester.py", line 332, in determine_test_result
    raise Exception(msg)

The diff and test file couls seem here https://gist.github.com/Daetalus/f5e0b27386b6752ae77b

Would you mind to tell me what I am missing or did wrong?

PS: Already made a clean build, not work...

Kevin Modzelewski
@kmod
Aug 04 2015 08:25
do you have a branch up on github?
Sun
@Daetalus
Aug 04 2015 08:25
Not push it yet.
Wait a moment please.
Kevin Modzelewski
@kmod
Aug 04 2015 08:46
ah, ok it's that there's a test called "future_builtins.py"
so when you do import future_builtins, you're getting the test file, not the actual module you wanted
this is not the first time that we've run into this :)
we have a check to make sure that you don't call your test file the same thing as a builtin module, but that only works if you run through make check_dbg or something similar, and it also doesn't catch conflicts with builtin c extensions
:/
Sun
@Daetalus
Aug 04 2015 08:51
Such a stupid mistake...
Thanks!
And does the CMakeLists and Makefile is correct?
Kevin Modzelewski
@kmod
Aug 04 2015 08:56
yeah looks right :)
you don't have to bother with the Makefile though, since that part of it is dead
(we should get around to removing all that stuff at some point)
Sun
@Daetalus
Aug 04 2015 08:56
Ok, thanks
Marius Wachtler
@undingen
Aug 04 2015 11:19
Took a brief look at the new GDB jit api but it's quite annoying:
  • plugin must be gpl
  • gdb headers not included in the ubuntu packages
  • I think the gdb plugin can't directly communicate with pyston. So the bjit would have to store the address ranges into a file (like the -p flag) and the gdb plugin would have to read it and tell gdb about it.
  • gdb must be told with jit-reader-load <path to .so> that it has to load the reader...
I think just emitting dummy ELF objects may even be better... because then we don't need the plugin detour which one has to load manually...
Maybe writing a dummy elf object isn't to hard if we can use a template and just change symbol name + address + size?
Marius Wachtler
@undingen
Aug 04 2015 16:15
Ok tried different routes, elf template, using LLVM MC layer. But the only solution I got to work is generating a assembler string with the correct sym name calling gcc to asm it and than registering this one with gdb.
While this is the ugliest route I think it should be fine for a debug use / special flag
Marius Wachtler
@undingen
Aug 04 2015 16:53
and now I also noticed that it's slow :-(
Sun
@Daetalus
Aug 04 2015 21:57
Hi @kmod #797 broken, I can't figure out why threading_local.py failed on clang debug mode.
Kevin Modzelewski
@kmod
Aug 04 2015 22:00
we have some transient issues with that test
I copied the stacktrace for us to debug and just restarted the travis-ci build
@rudi-c looks like it dies at an interesting part of the gc (asserting that a weakly_referenced object is a valid gc object)
Rudi Chen
@rudi-c
Aug 04 2015 22:03
On the bright side, it's a small test and not virtualenv.
I might be able to debug it by running it in a loop.
Sun
@Daetalus
Aug 04 2015 22:04
Maybe our str hash function need to rewrite? Please see the description of #797
Kevin Modzelewski
@kmod
Aug 04 2015 22:05
I don't think I have ever reproduced a threading_local crash
Rudi Chen
@rudi-c
Aug 04 2015 22:06
Yikes
Kevin Modzelewski
@kmod
Aug 04 2015 22:06
I wonder if there's something else that needs to happen too (ex needs to actually parse the file and not just load the cached version), or if there's some weird threading condition that causes it to hit an unusual ordering
Kevin Modzelewski
@kmod
Aug 04 2015 22:18
@daetalus yeah it sounds reasonable to switch to cpython's string hash
Kevin Modzelewski
@kmod
Aug 04 2015 23:06
man I hate performance investigations
I've narrowed this performance difference to... a documentation change
and the odd thing is that it's repeatable on another machine
Travis Hance
@tjhance
Aug 04 2015 23:07
like I said, pyston is an AI that reads its own documentation to figure out what it’s supposed to do
good documentation matters
Sun
@Daetalus
Aug 04 2015 23:10
@kmod CPython's string hash need the random.c. Should we add it to from_cpython?
Kevin Modzelewski
@kmod
Aug 04 2015 23:13
yeah, from_cpython/Python
are we running into anything that needs the hash randomization?
Sun
@Daetalus
Aug 04 2015 23:13
just test_hash need it.
For now.
Kevin Modzelewski
@kmod
Aug 04 2015 23:14
ah ok, was just curious, it's probably a good thing for us to have