These are chat archives for dropbox/pyston

29th
May 2015
Chris Toshok
@toshok
May 29 2015 00:04
for PyErr_NormalizeException, what’s the purpose of the traceback argument?
Rudi Chen
@rudi-c
May 29 2015 00:26
vagrant@vagrant-ubuntu-trusty-64:~/pyston$ python test/tests/pyc_stress_test.py
Traceback (most recent call last):
  File "test/tests/pyc_stress_test.py", line 37, in <module>
    os.remove(path)
OSError: [Errno 2] No such file or directory: 'test/tests/pyc_import_target.pyc'
Is this normal?
Chris Toshok
@toshok
May 29 2015 00:28
# Note: CPython doesn't pass this test
yeah :)
Rudi Chen
@rudi-c
May 29 2015 00:29
Oh right so in those cases the testing script only checks the exit code with Pyston?
Chris Toshok
@toshok
May 29 2015 00:30
that’d be my guess, yeah
Chris Toshok
@toshok
May 29 2015 00:45
weird, the incremental tracebacks help a ton in django/virtualenv, but they hurt slightly just about everything else in pyston-perf
Marius Wachtler
@undingen
May 29 2015 00:48
Mmmh I suspect that half of the tests don't even throw exceptions?!? (at least the old benchmarks)
Chris Toshok
@toshok
May 29 2015 00:49
Yeah, that's my impression too - but that should just mean no change. Some were 3-4%
Marius Wachtler
@undingen
May 29 2015 00:53
maybe it's some kind of constant startup overhead created by the site.py / codecs / etc import and the module inits which will get always executed before the main module?
and which may throw a lot of exceptions?
Chris Toshok
@toshok
May 29 2015 00:58
I think the more exceptions are thrown, the greater the benefit
Chris Toshok
@toshok
May 29 2015 04:39
interesting. it’s almost as if a raise of a pre-existing exception doesn’t include that location in the traceback
Kevin Modzelewski
@kmod
May 29 2015 05:40
how are you raising it?
Chris Toshok
@toshok
May 29 2015 05:41
actually i don’t think that’s it. there’s some weirdness going on. that contrived example in the PR shows both the raise3 + f(), in pairs, for each raise done
in cpython if I pass None for the traceback object instead of reusing, the raise line shows up in the output. if i reuse the traceback object, I don’t get any raise line (and certainly not one per iteration)
Chris Toshok
@toshok
May 29 2015 06:48
okay, yeah cpython treats a raise with a traceback as a re-raise
and re-raises don’t add a frame for the raise line
Daniel Agar
@dagar
May 29 2015 14:34
you could put all the test submodules in a different repo
Marius Wachtler
@undingen
May 29 2015 14:50
And add some travis ci script which fetches the other repo and runs the additional tests?
Daniel Agar
@dagar
May 29 2015 17:27
yep
or have cmake or the makefile do it
but optionally so everyone doesn't have to clone the entire thing
Chris Toshok
@toshok
May 29 2015 20:46
oh, huh. PyTraceback_Here is what cpython uses
I guess I’m misremembering.. must have been the api that cython used that took the string, that then called PyTraceback_Here with the fake frame
Rudi Chen
@rudi-c
May 29 2015 20:58
backedges are back edges in the CFG if we were to do a DFS from the starting point?
Kevin Modzelewski
@kmod
May 29 2015 20:59
hmm
the actual condition is that it's from a BB with a higher index to a lower index
but I think we do a DFS to generate the BBs from the AST, so maybe yes?
Rudi Chen
@rudi-c
May 29 2015 21:00
What's a BB?
Kevin Modzelewski
@kmod
May 29 2015 21:00
basic block
Rudi Chen
@rudi-c
May 29 2015 21:00
oh right
Kevin Modzelewski
@kmod
May 29 2015 21:00
err I guess we call them CFGBlocks in the codebase
Rudi Chen
@rudi-c
May 29 2015 21:48
28 instances of the string "del" in the server codebase, not too bad
Chris Toshok
@toshok
May 29 2015 21:48
nice
Rudi Chen
@rudi-c
May 29 2015 21:48
  • __del__
Moving finalizer calls to allowGLReadPreemption doesn't break any of the simple test, but unfortunately doesn't magically fix the integration tests.
Kevin Modzelewski
@kmod
May 29 2015 21:54
do you have frame introspection info on the calls to allowGLReadPreemption?
you want this line
Rudi Chen
@rudi-c
May 29 2015 21:58
What's frame introspection info and why might I need it for finalizer calls?
Kevin Modzelewski
@kmod
May 29 2015 21:59
well, in Python there are various ways to get information about the call stack
cpython, and our interpreter, actively maintain that information in a form that's easy to get at
but for code that goes through the llvm jit, we have to attach special metadata
(current line number, location of local variables, maybe more)
if we don't have that metadata and then we try to do something that would need it, things crash
currently, all exceptions look up the frame info for all parent frames (which chris is working on fixing), so any exception thrown in a finalizer will probably crash
even once chris fixes that, there are other ways to look at the frame info
Rudi Chen
@rudi-c
May 29 2015 22:05
Is that necessary for calls from the interpreter too?
The only call outside the interpreter is the one you're linking to.
Kevin Modzelewski
@kmod
May 29 2015 22:08
it should be ok from the interpreter
oh, but you have to rebase past that commit that I linked to
Rudi Chen
@rudi-c
May 29 2015 22:10
Oh yeah, already done that.
Rudi Chen
@rudi-c
May 29 2015 22:24
I'm thinking of also calling all finalizers in the pending list if the user calls gc.collect()
Do you guys think that makes sense?
Chris Toshok
@toshok
May 29 2015 22:27
i don’t have strong feelings for or against, really.. although other VM’s don’t do that
even on vm’s where gc.collect() actually causes a collection to happen (some just hint to the gc to do it sooner)
is this just to make tests easier to write?
we could add another entrypoint under gc (or probably under __pyston__), similar to .net’s GC.WaitForPendingFinalizers()
Rudi Chen
@rudi-c
May 29 2015 22:29
For now yes, because we don't have any APIs to force finalizer calls.
Should we eventually have a finalizer API?
Thinks like "disableFinalizerCallsUntil..."
*Things
Chris Toshok
@toshok
May 29 2015 22:30
i’d rather not. too easy for applications to think they’re doing things that make them more performant, causing hell for the gc :)
Rudi Chen
@rudi-c
May 29 2015 22:31
Yeah, since it won't be well-documented anyway (like the VMs for languages where the VM is an official one), I'd rather finalizers be abstracted away too.
Rudi Chen
@rudi-c
May 29 2015 22:51
pyc_stress_test is flaky for me (timeouts), does that happen to anybody else?
I guess I could just change the loop constant
Also is there a way to do something like "make check_integration"?
Chris Toshok
@toshok
May 29 2015 23:20
ugh, we’re passing ExcInfo by value to the catch block (and the unwinder depends on this)
there’s a bit of state we need to update there to deal with the re-raising vs. normal exception state
Marius Wachtler
@undingen
May 29 2015 23:25
oh.... the cheetah travis-ci test run which outputted: /bin/sh: 1: cheetah: not found
made me realize that I had the debian cheetah package installed on my system and I tested half of the time this one instead of the one installed inside virtualenv.... :-P
Chris Toshok
@toshok
May 29 2015 23:58
so removing 3 LogTimers from pyston::unwind_loop = 0.1 seconds
i’m thinking that machinery should use no RAII + dtors