These are chat archives for dropbox/pyston

29th
Apr 2015
Chris Toshok
@toshok
Apr 29 2015 00:13
c++ really needs alternative input methods
a foot pedal that inserts static_cast< for you
synecdoche
@synecdoche
Apr 29 2015 01:05
hmm. I think you could do that with the kinesis advantage and the optional foot pedal.
Travis Hance
@tjhance
Apr 29 2015 01:34
the tuples test is failing on master, for me
-((1, 2),)
+(1, 2)
it looks like a parsing thing, but I think my pypa is up-to-date
I have the pypa commit Fix parsing of '((1, 2),)’
Kevin Modzelewski
@kmod
Apr 29 2015 01:42
you probably have to blow away your pyc's
(maybe we should bump the pyc magic number)
Travis Hance
@tjhance
Apr 29 2015 01:43
wait, why?
is this a pyston pyc?
Kevin Modzelewski
@kmod
Apr 29 2015 01:43
yeah
so you might have cached the bad parse of the file
Travis Hance
@tjhance
Apr 29 2015 01:43
oh does it also cache the parse results?
where are they saved?
Kevin Modzelewski
@kmod
Apr 29 2015 01:44
just next to the .py file
Travis Hance
@tjhance
Apr 29 2015 01:45
maybe we should disable the pyc’s for the tests
this sounds likely to induce a lot of head-bashing
huh
now i’m getting
make lint
make[1]: Entering directory `/home/tjhance/pyston'
pyston: linting...
Lint checks passed
make[1]: Leaving directory `/home/tjhance/pyston'
make check_format
make[1]: Entering directory `/home/tjhance/pyston'
ninja -C /home/tjhance/pyston-build-release check-format
ninja: Entering directory `/home/tjhance/pyston-build-release'
ninja: error: loading 'build.ninja': No such file or directory
make[1]: *** [check_format] Error 1
make[1]: Leaving directory `/home/tjhance/pyston'
make: *** [check] Error 2
when I run make check
make check was working, like, 5 minutes ago
Kevin Modzelewski
@kmod
Apr 29 2015 01:50
ah ok I think make format doesn't work until you do make pyston_release first
Travis Hance
@tjhance
Apr 29 2015 01:51
make pyston_release fails too
CMake Error: cannot write to file "/home/tjhance/pyston-build-release/llvm/cmake/modules/CMakeFiles/Export/share/llvm/cmake/LLVMExports.cmake": No such file or directory
Kevin Modzelewski
@kmod
Apr 29 2015 01:53
it's hard to tell why that would happen without more context
you could try doing rm ~/pyston-build-release/build.ninja; make ~/pyston-build-release/build.ninja
and if that doesn't work you might want to just blow away your pyston-build-release directory
Travis Hance
@tjhance
Apr 29 2015 01:54
same thing
Kevin Modzelewski
@kmod
Apr 29 2015 01:54
maybe also rm -rf ~/pyston-build-release/{CMakeCache.txt,CMakeFiles}
Travis Hance
@tjhance
Apr 29 2015 01:55
I removed all of p yston-build-release and got the same thing
Kevin Modzelewski
@kmod
Apr 29 2015 01:55
is there a CMakeCache.txt or CMakeFiles/ in your ~/pyston directory?
Travis Hance
@tjhance
Apr 29 2015 01:55
no
ther’es also no ~/pyston-build-release/llvm
no i lied
there is
but there is no /pyston-build-release/llvm/cmake/modules/CMakeFiles/Export
well I have to go, I’ll figure this out later
Kevin Modzelewski
@kmod
Apr 29 2015 01:58
and this is after you did rm -rf ~/pyston-build-release?
Travis Hance
@tjhance
Apr 29 2015 01:58
right
Kevin Modzelewski
@kmod
Apr 29 2015 01:58
maybe move your ~/pyston dir and reclone
Travis Hance
@tjhance
Apr 29 2015 01:58
yeah i’ll try that later
does cmake use the pyston_deps at all?
Kevin Modzelewski
@kmod
Apr 29 2015 01:59
for llvm I think
Travis Hance
@tjhance
Apr 29 2015 02:04
But it uses the submodule for pypa and libunwind?
Chris Toshok
@toshok
Apr 29 2015 02:04
yes
and uses the system gcc iirc, not the one in pyston_deps
Travis Hance
@tjhance
Apr 29 2015 02:06
Why does it no longer need that?
Chris Toshok
@toshok
Apr 29 2015 02:10
Did we stop maintaining patches?
Travis Hance
@tjhance
Apr 29 2015 02:33
Patches? For gcc?
Chris Toshok
@toshok
Apr 29 2015 02:34
I can't remember if we ever had them..
Travis Hance
@tjhance
Apr 29 2015 02:34
I don't recall any
Chris Toshok
@toshok
Apr 29 2015 03:12
hrm, looks like generators defeat the whole RAII stat timer stack thing :/
I bet I can fix it just by stopping popping the timer in yield
Travis Hance
@tjhance
Apr 29 2015 05:40
I see all the cpython tests, and can run each individually, but is there a script that runs all of them?
Travis Hance
@tjhance
Apr 29 2015 05:45
ah, I found rntz’s NOTES
Marius Wachtler
@undingen
Apr 29 2015 08:16
ok the __eq__, __ne__ stuff is working for old style classes :-). Don't know what I did yesterday wrong / why they did not get called.
which means we pass now all 1890 tests of pycrypto :-)
Kevin Modzelewski
@kmod
Apr 29 2015 08:20
nice!
Marius Wachtler
@undingen
Apr 29 2015 08:23
Did you already find time to take a look at the virtualenv performance?
Kevin Modzelewski
@kmod
Apr 29 2015 08:27
a bit
I managed to eliminate a couple things -- it's not tracebacks or exceptions, it's not parsing, it's not jitting
I think it's the same stuff that's making django slow
(slowpath performance)
so I'm hoping Chris's work on that will improve virtualenv as well, or at least that the stats stuff he's adding will be helpful for virtualenv
this is definitely on the agenda for tomorrow :)
Marius Wachtler
@undingen
Apr 29 2015 08:29
:-)
yeah looking forward to having a better way to analyze the stats / have not only counters
Marius Wachtler
@undingen
Apr 29 2015 11:30
I'm getting anyone an idea? clfunc->versions is empty
python: src/codegen/irgen/hooks.cpp:576: void  pyston::CompiledFunction::speculationFailed(): Assertion `found' failed.
Chris Toshok
@toshok
Apr 29 2015 15:37
the stat timer overhead is pretty huge :/
from 9.1ms per iteration to 40.3ms per iteration on the django-template minibenchmark :/
Marius Wachtler
@undingen
Apr 29 2015 15:40
:-(
Chris Toshok
@toshok
Apr 29 2015 15:40
going to switch to rdtsc and see if I can massage it somehow
Marius Wachtler
@undingen
Apr 29 2015 15:41
Chris Toshok
@toshok
Apr 29 2015 15:41
hm, yeah. let me check out libstdc++ implements that
Marius Wachtler
@undingen
Apr 29 2015 15:42
ok rdtc would also work. I noticed before that our Timer class has a lot of overhead.
looking forward to testing your patch :-)
Chris Toshok
@toshok
Apr 29 2015 16:12
hm, std::high_resolution_clock uses clock_gettime, still a syscall it seems
it’s faster, though. from 40ms down to 30ms
rdtsc drops it back down to 9.9ms, but now to figure out how to reliably convert from ticks to us
Travis Hance
@tjhance
Apr 29 2015 16:20
Hey @kmod so here is the output from my make commands
$ make ext_pyston_selfhost
make: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make: /home/tjhance/pyston_deps/llvm-trunk-build/Debug+Asserts/bin/llvm-config: Command not found
ninja -C /home/tjhance/pyston-build-dbg pyston copy_stdlib copy_libpyston ext_pyston 
ninja: Entering directory `/home/tjhance/pyston-build-dbg'
[2/2] Linking CXX executable pyston
ln -sf /home/tjhance/pyston-build-dbg/pyston pyston_dbg
cd ./test/test_extension; DISTUTILS_DEBUG=1 time ../../pyston_dbg setup.py build
options (after parsing config files):
options (after parsing command line):
option dict for 'build' command:
  {}
running build
Distribution.get_command_obj(): creating 'build' command object
running build_ext
Distribution.get_command_obj(): creating 'build_ext' command object
building 'basic_test' extension
creating build/temp.linux2-2.7
gcc -pthread -fno-strict-aliasing -g -O3 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/tjhance/pyston/test/test_extension/../../from_cpython/Include -c -Werror=implicit-function-declaration basic_test.c -o build/temp.linux2-2.7/basic_test.o
creating build/lib.linux2-2.7
gcc -pthread -shared build/temp.linux2-2.7/basic_test.o -o build/lib.linux2-2.7/basic_test.pyston.so
building 'descr_test' extension
gcc -pthread -fno-strict-aliasing -g -O3 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/tjhance/pyston/test/test_extension/../../from_cpython/Include -c -Werror=implicit-function-declaration descr_test.c -o build/temp.linux2-2.7/descr_test.o
gcc -pthread -shared build/temp.linux2-2.7/descr_test.o -o build/lib.linux2-2.7/descr_test.pyston.so
building 'slots_test' extension
gcc -pthread -fno-strict-aliasing -g -O3 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/tjhance/pyston/test/test_extension/../../from_cpython/Include -c -Werror=implicit-function-declaration slots_test.c -o build/temp.linux2-2.7/slots_test.o
gcc -pthread -shared build/temp.linux2-2.7/slots_test.o -o build/lib.linux2-2.7/slots_test.pyston.so
0.83user 0.13system 0:01.00elapsed 96%CPU (0avgtext+0avgdata 58024maxresident)k
3064inputs+2520outputs (0major+32870minor)pagefaults 0swaps
cd ./test/test_extension; ln -sf build/lib.unknown-2.7/basic_test.pyston.so build/lib.unknown-2.7/descr_test.pyston.so build/lib.unknown-2.7/slots_test.pyston.so .
$ make check
make: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make: /home/tjhance/pyston_deps/llvm-trunk-build/Debug+Asserts/bin/llvm-config: Command not found
make lint
make[1]: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make[1]: Entering directory `/home/tjhance/pyston'
make[1]: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make[1]: /home/tjhance/pyston_deps/llvm-trunk-build/Debug+Asserts/bin/llvm-config: Command not found
pyston: linting...
Lint checks passed
make[1]: Leaving directory `/home/tjhance/pyston'
make check_format
make[1]: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make[1]: Entering directory `/home/tjhance/pyston'
make[1]: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make[1]: /home/tjhance/pyston_deps/llvm-trunk-build/Debug+Asserts/bin/llvm-config: Command not found
ninja -C /home/tjhance/pyston-build-release check-format
ninja: Entering directory `/home/tjhance/pyston-build-release'
[1/1] cd /home/tjhance/pyston/src && /home/tjhance/pyston/tools/check_format.sh /home/tjhance/pyston-build-release/llvm/./bin/clang-format
Format checks passed
make[1]: Leaving directory `/home/tjhance/pyston'
make ext_python ext_pyston pyston_dbg
make[1]: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make[1]: Entering directory `/home/tjhance/pyston'
make[1]: /home/tjhance/pyston_deps/llvm-trunk-build/Release+Asserts/bin/llvm-config: Command not found
make[1]: /home/tjhance/pyston_deps/llvm-trunk-build/Debug+Asserts/bin/llvm-config: Command not found
cd ./test/test_extension; python setup.py build
running build
running build_ext
CCACHE_CPP2=yes ccache /home/tjhance/pyston_deps/llvm-trunk-build/Release/bin/clang -pthread -g -Werror -Wreturn-type -Wall -Wno-sign-compare -Wno-unused -Isrc -Ifrom_cpython/Include -fno-omit-frame-pointer -Wextra -Wno-sign-compare -Wno-unused-parameter  -fPIC -Wimplicit -O2 -Ifrom_cpython/Include -Wno-missing-field-initializers -Wno-tautological-compare -Wno-type-limits -Wno-strict-aliasing -enable-tbaa -ferror-limit=10 -Qunused-arguments -fcolor-diagnostics -c test/test_extension/basic_test.c -o test/test_extension/basic_test.o
ccache: FATAL: /home/tjhance/pyston_deps/llvm-trunk-build/Release/bin/clang: execv returned (No such file or directory)
make[1]: *** [test/test_extension/basic_test.o] Error 1
make[1]: Leaving directory `/home/tjhance/pyston'
make: *** [check] Error 2
Chris Toshok
@toshok
Apr 29 2015 16:28
well, if these numbers are to be believed, slowpath performance isn’t what’s hurting us at all
Chris Toshok
@toshok
Apr 29 2015 16:50
i guess we can believe them
Travis Hance
@tjhance
Apr 29 2015 16:50
what is hurting us, then?
Chris Toshok
@toshok
Apr 29 2015 16:51
we don’t have timers covering everything. my numbers for a ~9ms iteration only add up to ~4.5ms, and that should include time spent running python/c extension code
Chris Toshok
@toshok
Apr 29 2015 17:01

it even appears to include gc time. for a run that did a gc on the last iteration:

gc_collections: 1
gc_collections_us: 13831
zzz_slowpath_getitem_us: 13856

and one that didn’t:

zzz_slowpath_getitem_us: 24
Chris Toshok
@toshok
Apr 29 2015 17:08
neat. i’m liking the rdtsc clock though. adding a timer to gc_alloc only increases runtime by 2ms
Chris Toshok
@toshok
Apr 29 2015 17:22
afaics there are only two things we can’t use the stat timers to measure: swapContext, and unwinding
Chris Toshok
@toshok
Apr 29 2015 17:38
oh, and stat overhead itself
Chris Toshok
@toshok
Apr 29 2015 17:46
ohhh, but we don’t start the timers until we hit a slowpath, so as long as we’ve rewritten things further up the stack, we wouldn’t have hit a slowpath and wouldn’t start the timer. but meanwhile we’re still running python + C extension code
Marius Wachtler
@undingen
Apr 29 2015 18:16
sounds interesting :-)
Kevin Modzelewski
@kmod
Apr 29 2015 19:40
@tjhance ah ok so the issue is we changed what we return for sys.platform
which changes the build directory that setup.py creates
if you replace lib.unknown-2.7 with lib.linux2-2.7 in the Makefile I think it should work
Travis Hance
@tjhance
Apr 29 2015 19:42
why is it only failing for me?
Kevin Modzelewski
@kmod
Apr 29 2015 19:42
it only fails if you've never built the makefile version
Marius Wachtler
@undingen
Apr 29 2015 19:46
looks like llvm really does not like that we have many operands on the patchpoints: when I run perf record on the whole system and execute the virtualenv test the hottest function is:
4,42% pyston_release pyston_release [.] llvm::MachineInstr::addOperand
Chris Toshok
@toshok
Apr 29 2015 19:47
okay PR #475 has the rdtsc clock in it now
Marius Wachtler
@undingen
Apr 29 2015 19:47
I mean it makes kind of sense that the optimized the code to be fast for instructions with a very low number of operands..
Chris Toshok
@toshok
Apr 29 2015 19:47
along with more timers (and an assert in callFunc to make sure we’ve got an active timer there)
Kevin Modzelewski
@kmod
Apr 29 2015 19:48
virtualenv_test compiles more than 500 functions, so you can try increasing the objectcache limit
Chris Toshok
@toshok
Apr 29 2015 19:48
@undingen yeah I’ve been seeing that function show up high in perf output for a while. never the hottest before though.
Kevin Modzelewski
@kmod
Apr 29 2015 19:48
I tried increasing it to 2500 and got 100% hit rate :)
but it didn't help the overall perf that much
(I think one of the reasons it needs more than 500 is because even though it uses a lot of the same functions they end up having different function names in different subprocesses)
Marius Wachtler
@undingen
Apr 29 2015 19:51
ok wil ltry
btw I finally fixed the pyxl / coding bug. Will upload it today but have to clean it up alittle bit
Kevin Modzelewski
@kmod
Apr 29 2015 20:06
@undingen hmm that speculationFailed() thing I guess is possible
maybe try changing >= 4 to == 4?
Marius Wachtler
@undingen
Apr 29 2015 20:07
will try
Kevin Modzelewski
@kmod
Apr 29 2015 20:07
I guess it's possible to hit that if there's a reentrant function, where two stack frames of it both trip the >= 4 threshold
Marius Wachtler
@undingen
Apr 29 2015 20:25
== 4fixed the issue. thanks
Michael Arntzenius
@rntz
Apr 29 2015 21:28
@toshok could you send me / link me the libgcc _Unwind_Find_FDE patch?
Chris Toshok
@toshok
Apr 29 2015 21:28
ah yeah, let me do that now
@kmod: so I added a rdtsc clock around the entirety of pyston::main(), and even that doesn’t add up to the total time the timers output. If I disable all timers except the StatTimer added to main, they’re very very close:
54923 logAndPop
main() rdtsc time = 54976
Kevin Modzelewski
@kmod
Apr 29 2015 21:34
@tjhance ok pushed dropbox/pyston@0256e16 which should hopefully get make check closer for you
Travis Hance
@tjhance
Apr 29 2015 21:37
yeah it seems to work now
at least, it gets to the part where it runs tests
thanks!!
Chris Toshok
@toshok
Apr 29 2015 22:13
@kmod I think I'm going to add endAt/restartAt methods to Timer so there will be zero gap (and only one rdtsc when we switch timers)
Marius Wachtler
@undingen
Apr 29 2015 22:24
Maybe we should keep track of the time in clockticks or nano seconds and only when stat dumping do the conversation to ms?
in order to not add up the errors?
Chris Toshok
@toshok
Apr 29 2015 22:26
Also a good call. That'll be harder given the way we tally directly into Stats on pause as well as end, but can definitely be done
Marius Wachtler
@undingen
Apr 29 2015 22:27
oh ok didn't look much at the code yet
Kevin Modzelewski
@kmod
Apr 29 2015 22:58
there's probably not much benefit to trying to reuse the Timer class
we could probably just do the timing directly in the StatTimer object?
that way we can also keep the Timer in us
Michael Arntzenius
@rntz
Apr 29 2015 23:17
uh-oh. I have a nondeterministic bug in collections_test under the unwinder
Kevin Modzelewski
@kmod
Apr 29 2015 23:21
try echo 0 | sudo tee /proc/sys/kernel/randomize_va_space
I mean, it's not great if we are sensitive to aslr, but at least it can help with debugging
Michael Arntzenius
@rntz
Apr 29 2015 23:22
aha, yeah
ah, that does seem to take the randomness out, but in this case it makes us always succeed, which is unfortunate
but maybe it'll make some other test deterministically fail
Michael Arntzenius
@rntz
Apr 29 2015 23:40
oh god, whether it fails depends on the precise length of the path passed to pyston_dbg. it succeeds if I use a short, relative path. it succeeds if I use a long, absolute path. but it fails if I use a long, absolute path with one or two extraneous "./" in it somewhere. but three extraneous "./"s? nope, then it succeeds.
it's probably just what this does to allocation & memory layout