These are chat archives for symengine/symengine

9th
Aug 2015
Ondřej Čertík
@certik
Aug 09 2015 04:29
@rwst it also depends on the benchmark. Some of the benchmarks use a lot of integers, and those will then incur an overhead if you have to use Python. I am trying to finish up a benchmark that uses PyDy, for some problems that people do (generating equations of motion for a bicycle, which are very long). I have it in SymEngine, but have to still do it in Sage.
Ralf Stephan
@rwst
Aug 09 2015 06:04
@isuruf 20 years ago they would have said google is your friend, but I'm getting soft these days ;) http://wiki.sagemath.org/ValgrindingSage also Titus Nicolae has done work which I adapted for this: https://github.com/titusnicolae/pynac-callgrind
Isuru Fernando
@isuruf
Aug 09 2015 06:11
Thanks, I'll take a look. It says to use a fresh copy and build, so that's going to take some time.
Just running callgrind on the python executable for benchmark S1 for symengine gives me 31% in symengine_wrapper.so not ignoring the initialization costs.
Ralf Stephan
@rwst
Aug 09 2015 07:05
@isuruf I did a double-check and I'm no longer sure about the ability of callgrind_control -ito control data collection. Can you please check too by simply starting sage --callgrind turn on/off control, then quit?
Ralf Stephan
@rwst
Aug 09 2015 07:14
Okay, the instructions on the wiki page are incomplete. With callgrind_control -iyou only turn on control. One needs -z to zero gathered data and --dump to get what's needed. Now Pynac is at 7.5% with symbench R2.
Ralf Stephan
@rwst
Aug 09 2015 08:06
The new numbers for Pynac are now: R1 1.3%, R2 7.5%, R3 4%, R6 0.09%, R7 7%, R8 0.8%, and for the first time S1 with a whopping libc 33%, Pynac 32%, libstdc++ 21%.
Isuru Fernando
@isuruf
Aug 09 2015 08:25
For S1, as mentioned earlier I get the same in symengine as well
Ralf Stephan
@rwst
Aug 09 2015 08:31

@isuruf What's your absolute time, for comparison?

sage: var('x,y,z')
sage: f = (x+y+z+1)^7
sage: %timeit g = expand(f*(f+1))
10 loops, best of 3: 39.4 ms per loop

(on a 3GHz AMD)

Isuru Fernando
@isuruf
Aug 09 2015 08:39
sage: var('x,y,z')
(x, y, z)
sage: f = (x+y+z+1)^7
sage: %timeit g = expand(f*(f+1))
10 loops, best of 3: 31.9 ms per loop
In [5]: from symengine import *

In [6]: var('x,y,z')
Out[6]: (x, y, z)

In [7]: f = (x+y+z+1)**7

In [8]: %timeit g = (f*(f+1)).expand()
100 loops, best of 3: 19.3 ms per loop
Ralf Stephan
@rwst
Aug 09 2015 08:46
In SymPy without symengine I get 54.2 µs. Is that caching?
Isuru Fernando
@isuruf
Aug 09 2015 08:46
Yes
Ralf Stephan
@rwst
Aug 09 2015 08:48
Nice, we don't have that as default in Sage, only for functions explicitly declared.
Isuru Fernando
@isuruf
Aug 09 2015 08:49
You can turn it off by setting SYMPY_USE_CACHE environment variable to no.
In [4]:  %timeit g = (f*(f+1)).expand()
1 loops, best of 3: 3.41 s per loop
SymPy's expand is very slow
Ralf Stephan
@rwst
Aug 09 2015 08:53
I will have a better look at symengine if Sage symbolic development stalls further due to missing reviewers.
Ondřej Čertík
@certik
Aug 09 2015 14:02
@rwst do you have some ideas that could make pynac faster for this benchmark?
Ralf Stephan
@rwst
Aug 09 2015 15:12
@certik I don't see the need. If people need fast polynomial manipulation in Sage they use one of the rings not symbolics. On this benchmark rings give a 10x speedup. I should have warned you that when I say symbolics this means Sage's symbolic "ring" that contains everything not fitting into the algebraic machinery, I.e. functions, solvers, symbols, calculus, expression manipulations, relations, see our ticket overview at http://trac.sagemath.org/wiki/symbolics
Ralf Stephan
@rwst
Aug 09 2015 15:18
You may ask why Sage doesn't 'simply switch to rings with pure polynomials. Conversion may eat the benefits, but this has never been fully addressed because algebraists are not interested in symbolics.
Ondřej Čertík
@certik
Aug 09 2015 16:45
@rwst sure, it's a synthetic benchmark, and since it's also a polynomial benchmark, you can always say to just use polynomials. (And we are working on making it easy for the user to use some good polynomial library for that, just like Sage does.) But the point of the benchmark for symbolics is that it measures how fast the symbolics are (no matter that it is a actually polynomial, so you can get 10x or more speedup by using a dedicated polynomial library). However, even better is to use a benchmark, that simply is not a polynomial. Here is an example: https://github.com/certik/symengine/pull/10#issuecomment-129199481, and Sage seems 6x slower. That uses symbolics (as it has square roots, sin, cos and other things) and you need to calculate a derivative. It's the bicycle benchmark that I was talking about above. That's why I was asking if you see any ways to make pynac faster, or if you do not see a need for it, even for this real problem.
@isuruf I installed the latest symengine in Sage using your spkg, and I am still getting the GMP warnings while linking: https://gist.github.com/certik/96583e143b657e71ae74#file-gistfile1-txt-L165, as you can see from the full output, Python seems to be found correctly. I wonder what causes it. We should fix it before posting to sage-devel.
Isuru Fernando
@isuruf
Aug 09 2015 16:48
@certik, yes, I am looking at it.
Ondřej Čertík
@certik
Aug 09 2015 16:51
Looks like there are a couple of problems. One is during linking of the tests, but those are ultimately not installed. The other problem is probably coming from this line: https://gist.github.com/certik/96583e143b657e71ae74#file-gistfile1-txt-L295

which removes the RPATH, and then the so library links against my systemwide libraries:

ondrej@hawk:~/ext/sage-6.8-x86_64-Linux(develop)$ ldd /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libsymengine.so 
/home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libsymengine.so: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libsymengine.so)
    linux-vdso.so.1 =>  (0x00007ffca65b1000)
    libgmpxx.so.4 => /usr/lib/x86_64-linux-gnu/libgmpxx.so.4 (0x00007f2ed0cc1000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f2ed0a4d000)
    libmpc.so.3 => /usr/lib/x86_64-linux-gnu/libmpc.so.3 (0x00007f2ed0835000)
    libmpfr.so.4 => /usr/lib/x86_64-linux-gnu/libmpfr.so.4 (0x00007f2ed05d9000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2ed02d5000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2ecffcf000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2ecfdb9000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2ecf9f4000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f2ed11fa000)

Though that by itself is not a problem, because Sage uses LD_LIBRARY_PATH. So one really needs to test this under ./sage -sh, as follows:

ondrej@hawk:~/ext/sage-6.8-x86_64-Linux(develop)$ ./sage -sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/home/ondrej/ext/sage-6.8-x86_64-Linux
(sage-sh) ondrej@hawk:sage-6.8-x86_64-Linux$ ldd /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libsymengine.so
    linux-vdso.so.1 =>  (0x00007ffcc8b47000)
    libgmpxx.so.4 => /usr/lib/x86_64-linux-gnu/libgmpxx.so.4 (0x00007faaf6f33000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007faaf6cbf000)
    libmpc.so.3 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libmpc.so.3 (0x00007faaf6aa7000)
    libmpfr.so.4 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libmpfr.so.4 (0x00007faaf684d000)
    libstdc++.so.6 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib64/libstdc++.so.6 (0x00007faaf6543000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007faaf623d000)
    libgcc_s.so.1 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib64/libgcc_s.so.1 (0x00007faaf6027000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007faaf5c62000)
    libgmp.so.16 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libgmp.so.16 (0x00007faaf59d1000)
    /lib64/ld-linux-x86-64.so.2 (0x00007faaf746c000)

Now you can see that MPC, MPFR, GCC and other libraries are all linked properly against Sage, except GMP. So there is still a bug.

It's weird though, I don't know what might be causing this. The CMake output looks good and finds everything in Sage.
Isuru Fernando
@isuruf
Aug 09 2015 16:57
you only have a system-wide gmp right? no mpfr or mpc?
Ondřej Čertík
@certik
Aug 09 2015 17:04
Correct.
I opened #580 for the bicycle benchmark.
@isuruf on my computer with the #580 benchmark, Sage is 7x slower. But since symengine for some reason uses system-wide GMP, I need to rerun it after we make sure symengine is using the exact same libraries as Sage does.
Isuru Fernando
@isuruf
Aug 09 2015 17:19
@certik, this is a problem with cmake.
Ondřej Čertík
@certik
Aug 09 2015 17:21
Do you now what the problem is?
Isuru Fernando
@isuruf
Aug 09 2015 17:22
Take a look at the last paragraph at http://www.cmake.org/Wiki/CMake_2.6_Notes
Since gmp is also present as a system-wide directory, it is linked with only -l, but what we really want is to link with the full path
Take a look at a link.txt in the test directory. It should be something like,
/projects/b1cc51a6-c4a2-4281-98b4-a5c2b1dea21b/ext/sage-6.8-x86_64-Linux/local/bin/g++   -std=c++0x -Wall -Wextra -fPIC -O3 -march=native -ffast-math -funroll-loops -Wno-unused-parameter    CMakeFiles/test_basic.dir/test_basic.cpp.o  -o test_basic -rdynamic ../../libsymengine.so ../../catch/libcatch.a ../../libsymengine.so ../../teuchos/libteuchos.a -lmpc -lmpfr -lgmpxx -lgmp -Wl,-rpath,/projects/b1cc51a6-c4a2-4281-98b4-a5c2b1dea21b/ext/sage-6.8-x86_64-Linux/local/var/tmp/sage/build/symengine-0.1/src/symengine
Ondřej Čertík
@certik
Aug 09 2015 17:27
if I build from spkg, do you know if Sage leaves the build behind?
Isuru Fernando
@isuruf
Aug 09 2015 17:28
If it's successful, it will not. (There's an environment variable to be set to keep the build directory behind)
Ondřej Čertík
@certik
Aug 09 2015 17:29
I am rebuilding with VERBOSE=1 and see what happens.
Isuru Fernando
@isuruf
Aug 09 2015 17:29
setting SAGE_KEEP_BUILT_SPKGS to yes, will not remove the build directory
Ondřej Čertík
@certik
Aug 09 2015 17:30
Thanks. Do you think we need to use IMPORTED in cmake?
Isuru Fernando
@isuruf
Aug 09 2015 17:30
I just tried with the solution in the wiki page. (IMPORTED targets). Now instead of -lgmp and -lgmpxx, I get /projects/sage/sage-6.7/local/lib/libgmpxx.so /projects/sage/sage-6.7/local/lib/libgmp.so.
Ondřej Čertík
@certik
Aug 09 2015 17:32
It's build as follows:
/home/ondrej/ext/sage-6.8-x86_64-Linux/local/bin/g++   -std=c++0x -Wall -Wextra -fPIC -O3 -march=native -ffast-math -funroll-loops -Wno-unused-parameter    CMakeFiles/test_printing.dir/test_printing.cpp.o  -o test_printing -rdynamic ../../libsymengine.so ../../catch/libcatch.a ../../libsymengine.so ../../teuchos/libteuchos.a -lgmpxx -lgmp -lmpc -lmpfr -Wl,-rpath,/home/ondrej/ext/sage-6.8-x86_64-Linux/symengine-0.1/src/symengine
Which I think is incorrect. It should give the -L flag to Sage
Isuru Fernando
@isuruf
Aug 09 2015 17:33
yeah
Ondřej Čertík
@certik
Aug 09 2015 17:33
Something went wrong in cmake. I think this is unrelated to IMPORTED.
I would think that if it finds a given library (in Sage), that it will use it properly (either full path, or using -L and -l).
Ondřej Čertík
@certik
Aug 09 2015 17:38
Hm, so cmake somehow takes the full path (in Sage), and then happily links against systemwide...
I wonder if instead of our COMMON_DIR approach, we rather should be using CMAKE_PREFIX_PATH.
Isuru Fernando
@isuruf
Aug 09 2015 17:42
Even then, if a user wants to specify a library in a non standard location, it will link with the system one
Ondřej Čertík
@certik
Aug 09 2015 17:46

I figured out a fix for now --- add this before cmake in spkg-install:

export LDFLAGS="-L$SAGE_LOCAL/lib"

Now it seems to work.

(sage-sh) ondrej@hawk:symengine-0.1$ ldd ../local/lib/libsymengine.so 
    linux-vdso.so.1 =>  (0x00007fff8b0c0000)
    libgmpxx.so.8 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libgmpxx.so.8 (0x00007f67c3ea4000)
    libgmp.so.16 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libgmp.so.16 (0x00007f67c3c13000)
    libmpc.so.3 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libmpc.so.3 (0x00007f67c39fb000)
    libmpfr.so.4 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib/libmpfr.so.4 (0x00007f67c37a1000)
    libstdc++.so.6 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib64/libstdc++.so.6 (0x00007f67c3497000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f67c3191000)
    libgcc_s.so.1 => /home/ondrej/ext/sage-6.8-x86_64-Linux/local/lib64/libgcc_s.so.1 (0x00007f67c2f7b000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f67c2bb6000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f67c43db000)
I am just surprised and disappointed that cmake doesn't do this automatically, since it knows that we want to link against Sage's library (that we found), and not the systemwide.
Here is the patch: symengine/symengine.spkg#3
Ondřej Čertík
@certik
Aug 09 2015 17:52
Ok, looks like CMake changed this recently: http://www.cmake.org/cmake/help/v2.8.10/cmake.html#policy%3aCMP0003
Isuru Fernando
@isuruf
Aug 09 2015 17:54
That's not the reason. That change is good, since if we are giving the full path to a library, other libraries in that folder are not found.
The reason behind this issue is that when it is a system library -l is used.
Ondřej Čertík
@certik
Aug 09 2015 17:55
You are right. Still I don't understand why.
CMake takes our full path to gmp, and then discovers a system library, so then it changes full path to the -lgmp?
Isuru Fernando
@isuruf
Aug 09 2015 17:56
Yes
Ondřej Čertík
@certik
Aug 09 2015 17:57
Wow, that's completely broken....
cmake should only do this if the full path starts with /usr/lib
How complicated is the IMPORTED patch, that you tried?
I think that's the way to go.
Isuru Fernando
@isuruf
Aug 09 2015 17:59
You add
add_library(gmp STATIC IMPORTED)
set_property(TARGET gmp PROPERTY IMPORTED_LOCATION ${GMP_LIBRARY})
add_library(gmpxx STATIC IMPORTED)
set_property(TARGET gmpxx PROPERTY IMPORTED_LOCATION ${GMPXX_LIBRARY})
set(LIBS ${LIBS} gmpxx gmp)
Ondřej Čertík
@certik
Aug 09 2015 18:01
I see. I think this must be the way to go.
Thanks for figuring it out. We'll have to change our cmake files to use imported.
@isuruf why do you think cmake by default would do that?
Isuru Fernando
@isuruf
Aug 09 2015 18:07
They do it because they assume that find_library may return a wrong path. For example, find_library might return a math library for a different architecture. To avoid that it is changed in to -lm so that the linker will find it for you
Isuru Fernando
@isuruf
Aug 09 2015 18:45
I opened #581 to fix this
Ondřej Čertík
@certik
Aug 09 2015 19:09
I see. In our case, we print the paths, so that the user can check those. Then we want to use those.
I like your fix, but I need to look at it more carefully.
Isuru Fernando
@isuruf
Aug 09 2015 19:11
Sure