These are chat archives for dropbox/pyston

28th
Apr 2015
Chris Toshok
@toshok
Apr 28 2015 01:00
so given these numbers, these are the top three getattr calls: type.__call__, type.__class__, type.__instancecheck__
the latter two come from PyObject_IsInstance
Travis Hance
@tjhance
Apr 28 2015 04:01
This message was deleted
This message was deleted
This message was deleted
Travis Hance
@tjhance
Apr 28 2015 04:08
This message was deleted
This message was deleted
Chris Toshok
@toshok
Apr 28 2015 04:44
hm, slowpath timings are depressingly flat
Travis Hance
@tjhance
Apr 28 2015 04:44
what do you mean by ‘flat’?
flat over the past few months?
Chris Toshok
@toshok
Apr 28 2015 04:46
just no very clear hot spots
actually the hottest one is nonzero. odd
Travis Hance
@tjhance
Apr 28 2015 04:47
nonzero is called implicitly a lot in standard python code, right?
Chris Toshok
@toshok
Apr 28 2015 04:48
yeah, if tests, etc
toshok @toshok runs with more requests
Travis Hance
@tjhance
Apr 28 2015 04:50
lolwut
Chris Toshok
@toshok
Apr 28 2015 04:50
hoping we just haven’t rewritten enough
Travis Hance
@tjhance
Apr 28 2015 04:51
i’ve always been kind of worried that the frequent if (rewrite_args != NULL) checks would be a big slowdown for the slowpath
but I didn’t have any good ideas for how to reduce them
Chris Toshok
@toshok
Apr 28 2015 04:52
zzz_slowpath_createBoxedIterWrapperIfNeeded_us: 2                                    
zzz_slowpath_len_us: 4                                                               
zzz_slowpath_augbinop_us: 14                                                         
zzz_slowpath_getglobal_us: 41                                                        
zzz_slowpath_binop_us: 310                                                           
zzz_slowpath_setitem_us: 318                                                         
zzz_slowpath_setattr_us: 395                                                         
zzz_slowpath_compare_us: 1562                                                        
zzz_slowpath_callattr_us: 2152                                                       
zzz_slowpath_getitem_us: 2479                                                        
zzz_slowpath_nonzero_us: 2643                                                        
zzz_slowpath_getattr_us: 2695                                                        
zzz_slowpath_isinstance: 3321                                                        
zzz_slowpath_runtimecall_us: 5074
nothing really surprising (or ultimately all that helpful :/)
yeah, i’d love to figure out a way to split the fastpaths and the multiple bailout points somehow that they weren’t so interleaved
Travis Hance
@tjhance
Apr 28 2015 04:54
one thing I considered was compiling objmodel.cpp twice with different macro definitions
one of which makes all the rewriting stuff into no-ops
actually
maybe templates instead of macros
I’m thinking too C
e.g. give each function a template parameter RewritingEnabled
Chris Toshok
@toshok
Apr 28 2015 05:28
templates are definitely preferable yeah
but what about a template that exposes all the rewrite apis
the non-rewriting case would just have no-ops for all those apis
if that would even work
Travis Hance
@tjhance
Apr 28 2015 05:42
hm i guess if we replaced all the rewriter_args != NULL with some call to this template parameter, we could have it return true and then it gets inlined...
although it would also be nice if, for the sake of our sanity, we could avoid writing so many of these if-checks in the first place
Chris Toshok
@toshok
Apr 28 2015 05:44
yeah I’d love a single path that’s totally specialized based on template args, no if (rewriter_args != NULL)
i wonder if we could just abort a rewrite by throwing an exception
Travis Hance
@tjhance
Apr 28 2015 05:45
yeah but you also need to finish the execution
and it seems sort of silly to re-do work by starting over on the other path
Chris Toshok
@toshok
Apr 28 2015 05:51
right, each stage would be something like try { return getattrInternal<Rewritable>(…); } catch (RewriteAborted e) { return getattrInternal<Slowpath>(…); }
Travis Hance
@tjhance
Apr 28 2015 05:52
what if getattrInternal has already done some side-effectful thing (e.g., call __get__)
Chris Toshok
@toshok
Apr 28 2015 05:53
hm, yeah that would have to be expressed along the spine of things.. that operation would have a try/catch too
maybe? :)
Travis Hance
@tjhance
Apr 28 2015 05:53
like you need to be able to pick up where you left off
Chris Toshok
@toshok
Apr 28 2015 05:53
right
Travis Hance
@tjhance
Apr 28 2015 05:54
maybe the Rewritable case aborts but still finishes
Chris Toshok
@toshok
Apr 28 2015 05:59
hm, yeah then we’d need if’s anyway
we should just CPS the entire rewrite stack :)
Kevin Modzelewski
@kmod
Apr 28 2015 06:02
we could probably test this to see how much it could help
by running in -I mode, and then manually changing all the rewriter != NULL comparisons to just false
and seeing how much perf changes
Travis Hance
@tjhance
Apr 28 2015 06:04
well we wouldn’t necessarily need the ifs if we fold them into the rewriter api
even if perf doesn’t help, I think it could make the code a lot cleaner and easier to maintain
Marius Wachtler
@undingen
Apr 28 2015 08:08
do we already have a define inside the C-API headers for checking if it's pyston or cpython from an extension?
Kevin Modzelewski
@kmod
Apr 28 2015 09:27
_PYSTON_API is supposed to serve that purpose
though it should probably be renamed
and right now it's determined just by checking if we're compiling in C++11 mode :P
Marius Wachtler
@undingen
Apr 28 2015 09:39
and if set it let's PyTypeObject be c++ class.... I'm looking for some define which I can use to check inside pycrypto if it's including the pyston headers in order to know if I can use the pyston specifig GMP conversation routines.
currently I'm checking for the PYSTON_EXTINCLUDE_PYTHON_H header guard of Python.h :-D
Which works but probably should't be the recommended way to check it
Kevin Modzelewski
@kmod
Apr 28 2015 10:02
oh sorry, you mean for the extension code to determine which implementation it's targeting?
I just dealt with the other way (whether we are compiling pyston code with our headers or extension code with our headers) so I assumed you meant that
cython looks at PYPY_VERSION to see if it's pypy
it also checks Py_PYTHON_H (cpython's include guard)
maybe we could use PYSTON_VERSION?
Marius Wachtler
@undingen
Apr 28 2015 12:43
I'm looking forward to our custom c++ exception unwinder + tracebacks... the pycrypto testsuite needs currently 37secs to run and 10secs are spend in getraceback...
and this is without the gcc change. _Unwind_Find_FDE is showing up a lot so expect this can be significantly speed up.
cpythons needs 10/22secs (not sure why so much variance?? maybe the random number generator is waiting for randomness??)
synecdoche
@synecdoche
Apr 28 2015 19:53
hi. i'm getting a build error on Mint 17, which i'm hoping won't be too different from Ubuntu 14.04 to matter. any ideas?
FAILED: CCACHE_CPP2=yes /usr/bin/ccache /usr/bin/clang++   -DDEFAULT_PYTHON_MAJOR_VERSION=2 -DDEFAULT_PYTHON_MICRO_VERSION=6 -DDEFAULT_PYTHON_MINOR_VERSION=7 -DLLVMREV=230300 -DNVALGRIND -DTHREADING_USE_GIL=1 -DTHREADING_USE_GRWL=0 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -Qunused-arguments -fcolor-diagnostics  -Qunused-arguments -fcolor-diagnostics -Wall -Wextra -Werror -Wreturn-type -Wno-sign-compare -Wno-unused -Wno-sign-compare -Wno-unused-parameter -fno-omit-frame-pointer -std=c++11 -fno-rtti -fexceptions -fvisibility-inlines-hidden -ffunction-sections -fdata-sections -Woverloaded-virtual -Wno-invalid-offsetof -Wcast-qual -Wno-sign-conversion -Wnon-virtual-dtor -Winit-self -Wmissing-include-dirs -Wstrict-overflow=5 -Wpointer-arith -Wtype-limits -Wwrite-strings -Wempty-body -Waggregate-return -Wmissing-field-initializers -Wredundant-decls -Winline -Wint-to-pointer-cast -Wlong-long -Wvla -Wno-attributes -Qunused-arguments -fcolor-diagnostics -Wimplicit-int -Wstrict-prototypes -Wold-style-definition -Wnested-externs -Wpointer-to-int-cast -Wno-mismatched-tags -Wno-extern-c-compat -g -DBINARY_SUFFIX= -DBINARY_STRIPPED_SUFFIX=_stripped -Isrc -I/home/synec/pyston/src -I/home/synec/pyston_deps/llvm-trunk/include -Illvm/include -I/home/synec/pyston/from_cpython/Include -I. -Ilibunwind/include -I/home/synec/pyston/libpypa/src -I/home/synec/pyston/lz4/lib -MMD -MT src/CMakeFiles/PYSTON_OBJECTS.dir/codegen/entry.cpp.o -MF "src/CMakeFiles/PYSTON_OBJECTS.dir/codegen/entry.cpp.o.d" -o src/CMakeFiles/PYSTON_OBJECTS.dir/codegen/entry.cpp.o -c /home/synec/pyston/src/codegen/entry.cpp
In file included from /home/synec/pyston/src/codegen/entry.cpp:29:
In file included from /home/synec/pyston_deps/llvm-trunk/include/llvm/IR/IRBuilder.h:26:
In file included from /home/synec/pyston_deps/llvm-trunk/include/llvm/IR/LLVMContext.h:21:
In file included from /home/synec/pyston_deps/llvm-trunk/include/llvm/Support/Options.h:41:
/home/synec/pyston_deps/llvm-trunk/include/llvm/Support/CommandLine.h:458:8: error: 'llvm::cl::OptionValue<cl::boolOrDefault>' has virtual functions but non-virtual destructor [-Werror,-Wnon-virtual-dtor]
struct OptionValue<cl::boolOrDefault> final
       ^
there are lots more errors on CommandLine.h, but i'm trying to keep it brief.
Marius Wachtler
@undingen
Apr 28 2015 20:32
are you using gcc or clang to compile pyston?
and what compiler version
synecdoche
@synecdoche
Apr 28 2015 20:39
I believe it's building using clang 3.4. I just pulled the code, so it should be using whatever the standard defaults are when following the instructions in the CMake section of INSTALLING.md
synec@lamiaceae ~/pyston $ clang --version
Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4)
Target: x86_64-pc-linux-gnu
Thread model: posix
I also get these build errors right at the start of the build:
make: /home/synec/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make: /home/synec/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make: /home/synec/pyston_deps/llvm-trunk-build/Debug+Asserts/bin/llvm-config: Command not found
Marius Wachtler
@undingen
Apr 28 2015 20:42
mmh sorry I'm not sure whats wrong :-(. Maybe run make llvm_up
synecdoche
@synecdoche
Apr 28 2015 20:44
hmm. I hadn't run that before. it seems to have done things. I'll try make again. thanks :)
Marius Wachtler
@undingen
Apr 28 2015 20:47

@kmod

instance_cls->giveAttr("__eq__", new BoxedFunction(boxRTFunction((void*)instanceEq, UNKNOWN, 2)));
instance_cls->giveAttr("__ne__", new BoxedFunction(boxRTFunction((void*)instanceNe, UNKNOWN, 2)));

won't get called :-(, what am I doing wrong?

@synecdoche let me know if it works. if it's required after a fresh checkout we may want to update the docs or better make sure the build systems does the update on it's own. We recently switched the default to cmake (or the cmake + make hybrid) so there may be bugs hiding.
synecdoche
@synecdoche
Apr 28 2015 20:56
will do. looks like make llvm_up applied a bunch of patches, and llvm is rebuilding now.
synecdoche
@synecdoche
Apr 28 2015 21:15
@undingen hey hey, that seems to have worked! so yes, make llvm_up was the missing command.
Marius Wachtler
@undingen
Apr 28 2015 21:16
ok cool
synecdoche
@synecdoche
Apr 28 2015 21:16
:)
synec@lamiaceae ~/pyston $ ./pyston_dbg 
Pyston v0.3 (rev 2c7e7a5e0ed7a0a8b4528919f855fa8336b43902), targeting Python 2.7.6
>>
Marius Wachtler
@undingen
Apr 28 2015 21:17
:+1:
synecdoche
@synecdoche
Apr 28 2015 21:17
now let's try make check...
Marius Wachtler
@undingen
Apr 28 2015 21:17
@dagar Do you know why @synecdoche had to manually run make llvm_up after the initial checkout?
synecdoche
@synecdoche
Apr 28 2015 21:19
also, it looks like dbg is built by default, but make check tries to run against release
Michael Arntzenius
@rntz
Apr 28 2015 21:19
@synecdoche make check runs dbg and release
Chris Toshok
@toshok
Apr 28 2015 21:19
make check should rebuild as needed
Michael Arntzenius
@rntz
Apr 28 2015 21:23
hm, does the -n flag not force us also to JIT eval and execed code?
oh, it looks like it does, but that it's not getting _U_dyn_registered
Daniel Agar
@dagar
Apr 28 2015 21:26
make llvm_up should still be part of the instructions
and it's not...
Michael Arntzenius
@rntz
Apr 28 2015 21:28
oh, I see what's happening
Travis Hance
@tjhance
Apr 28 2015 21:29
we always enter eval or exec’ed code through the interpreter at the moment
we just don’t support doing it through the JIT right now, so it ignores -n
it should be able to OSR up, though
and functions declared inside the eval/exec should be able to JIT
Michael Arntzenius
@rntz
Apr 28 2015 21:31
oh, ok.
damn. I was looking for a good way to deliberately clog up the compiled objects list so I could test libunwind's equivalent of _Unwind_Find_FDE
Travis Hance
@tjhance
Apr 28 2015 21:33
it’s because our calling convention doesn’t support passing in a locals dictionary, for example
and you have to use the calling convention to enter a JITed function
whereas it’s easy to cheat the convention through the interpretter
Michael Arntzenius
@rntz
Apr 28 2015 21:34
hm. oh! I think we are talking about different things.
exec """def f(x): return x""" <- f will get JITted when I call it, right?
Travis Hance
@tjhance
Apr 28 2015 21:34
yes
Michael Arntzenius
@rntz
Apr 28 2015 21:34
ok, that'll do then
Travis Hance
@tjhance
Apr 28 2015 21:35
actually
we may have disabled that too
because globals :\
Michael Arntzenius
@rntz
Apr 28 2015 21:36
hm. so with verbose output on, it claims "JIT'ing <string>:a with signature ..."
Travis Hance
@tjhance
Apr 28 2015 21:36
but if you aren’t passing in globals, it will probably still work?
Michael Arntzenius
@rntz
Apr 28 2015 21:36
I'm not passing in globals
and it shows me a CFG, but not LLVM IR
yeah, it looks like it doesn't actually send it through LLVM :(
Chris Toshok
@toshok
Apr 28 2015 21:39
man, we allocate 1.5M to create 1 django Template object
actually that’s just calls to gc_alloc...
realloc doesn’t add much, 1586157 bytes
Marius Wachtler
@undingen
Apr 28 2015 21:50
I think you need pass -v -v to see the LLVM IR (if it get's created)
Kevin Modzelewski
@kmod
Apr 28 2015 21:57
it may say "JIT'ing ... at effort level 0" which is for the interpreter
Marius Wachtler
@undingen
Apr 28 2015 21:57
My pycrypto pull request just broke the gold linker? https://travis-ci.org/dropbox/pyston/jobs/60443119
Kevin Modzelewski
@kmod
Apr 28 2015 21:57
we never updated that message after adding the ast interpreter :/
but functions defined inside execs will not get jit'd either
Marius Wachtler
@undingen
Apr 28 2015 21:58
gold: internal error in find_view, at ../../gold/fileread.cc:334
Chris Toshok
@toshok
Apr 28 2015 21:58
wow
Travis Hance
@tjhance
Apr 28 2015 23:35
proposal: create a one-letter function to replace static_cast
I’m really tired of typing it out
This message was deleted