symengine/symengine

Fast symbolic manipulation library, written in C++

Isuru Fernando

@isuruf

pickles are supposed to be backward compatible

and independent of platform

if you pickle in python 2 on osx, you can unpickle in python 3 on linux

you can't pickle in python3 on osx and unpickle in python 2 on linux

so, pickle is not forward compatible, but if 2 people used the same python minor version, they'll be compatible. If we use boost, there's an additional constraint on boost version

Richard Otis

@richardotis

One use case where you're less sensitive to environment compatibility is using dask or multiprocessing, where serialization is basically just IPC

Though I very much understand that you don't want to implement serialization with a footgun that users might not notice

Richard Otis

@richardotis

Are the Windows symengine packages on conda-forge built with LLVM enabled?

Isuru Fernando

@isuruf

Yes. And wheels too

Richard Otis

@richardotis

And it's built with MSVC?

Richard Otis

@richardotis

I am trying to link to symengine.lib using the g++ that comes with the mingw-w64 package. It finds it, but I'm getting all undefined references during the compile step.

Isuru Fernando

@isuruf

Yes, they are built with MSVC

Richard Otis

@richardotis

Okay, looks like I'll need to try converting my toolchain over. Thanks.

Isuru Fernando

@isuruf

To use with mingw-w64, you'll have to use the C interface instead of C++

Richard Otis

@richardotis

Do I have any control over that from the Cython side?

Isuru Fernando

@isuruf

Sure. You can use the functions in cwrapper.h instead of the C++ classes

Richard Otis

@richardotis

That would mean recreating and vendoring the portion of symengine_wrapper I'm using?

Isuru Fernando

@isuruf

Yes, mostly the LLVMDoubleVisitor related functions

Isuru Fernando

@isuruf

@richardotis, @bocklund I looked at the pycalphad PR. One thing I saw was that you were building lots of LLVMDoubleVisitors. Are they called with the same input? If so, you might be missing out on cse optimizations.

Richard Otis

@richardotis

They usually are, but we can't guarantee that due to the structure of our optimization problem.

Richard Otis

@richardotis

We do some grouping of our constraint functions, which does get the CSE benefit

We are SymEngine and LLVM newbies though, if there are any new superpowers we may have gained, please let us know. ;)

Isuru Fernando

@isuruf

Can you post the symbolic expressions for something like mass_hess?

Richard Otis

@richardotis

We could probably pickle the sympy versions of the objects

Isuru Fernando

@isuruf

yeah, that works

python >3.4

Richard Otis

@richardotis

@bocklund probably has a good one for a complex system. With the latest update we're hoping to push beyond the limits of what the commercial codes can do in terms of function complexity

Isuru Fernando

@isuruf

For something like mass_hess, it might be beneficial to have a loop with the data in column major format to get SIMD benefits

Brandon Bocklund

@bocklund

Pickling and unpickling the LLVMDouble does not correctly reset the LLVMDoubleVisitor that is supposed to be in llvm_double[0].

Reproducing the bug

from symengine import symbols
x, y, z = symbols('x y z')
ex = x**2 + y**2 + z**2
from symengine import lambdify
l = lambdify([x, y, z], [ex], backend='llvm')
import pickle
print('pickle.loads(pickle.dumps(l)) == l  :: ', pickle.loads(pickle.dumps(l)) == l)
print(l.__reduce__())
print(pickle.loads(pickle.dumps(l)).__reduce__())

gives

pickle.loads(pickle.dumps(l)) == l  ::  False
(<built-in function llvm_loading_func>, (3, 1, [()], True, 1, 'C', [0, 1], <class 'numpy.float64'>, b"\xcf\xfa\xed\xfe\x07\x00\x00\x01\x03\x00\x00\x00\x01\x00\x00\x00\x04\x00\x00\x00`\x01\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x19\x00\x00\x00\xe8\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x00\x80\x01\x00\x00\x00\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x00\x07\x00\x00\x00\x07\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00__text\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00__TEXT\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'\x00\x00\x00\x00\x00\x00\x00\x80\x01\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00__eh_frame\x00\x00\x00\x00\x00\x00__TEXT\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00(\x00\x00\x00\x00\x00\x00\x008\x00\x00\x00\x00\x00\x00\x00\xa8\x01\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0b\x00\x00h\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00$\x00\x00\x00\x10\x00\x00\x00\x00\x0e\n\x00\x00\x00\x00\x00\x02\x00\x00\x00\x18\x00\x00\x00\xe0\x01\x00\x00\x01\x00\x00\x00\xf0\x01\x00\x00\x10\x00\x00\x00\x0b\x00\x00\x00P\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf2\x0f\x10\x07\xf2\x0f\x10O\x08\xf2\x0f\x10W\x10\xf2\x0fY\xd2\xf2\x0fY\xc0\xf2\x0fX\xc2\xf2\x0fY\xc9\xf2\x0fX\xc8\xf2\x0f\x11\x0e\xc3\x00\x14\x00\x00\x00\x00\x00\x00\x00\x01zR\x00\x01x\x10\x01\x10\x0c\x07\x08\x90\x01\x00\x00\x1c\x00\x00\x00\x1c\x00\x00\x00\xb8\xff\xff\xff\xff\xff\xff\xff'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x0e\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00___unnamed_1\x00\x00\x00"))
(<built-in function llvm_loading_func>, (3, 1, [()], True, 1, 'C', [0, 1], <class 'numpy.float64'>, b''))

You can see that the pickled/unpickled version gives an empty byte string for the dumped LLVMDoubleVisitor bytes. @richardotis and I were tracking this down and the issue looks to be that the membuffer is not correctly reset in LLVMDoubleVisitor.loads here when called in self.llvm_double[0].loads as part of the LLVMDouble._load method here. In that case, the root fix would be to set the membuffer in LLVMDoubleVisitor.loads in the same way as LLVMDoubleVisitor.__init__.

A test for this might be to do pickle.loads(pickle.dumps(pickle.loads(pickle.dumps(l)))), which currently will kill the kernel with a LLVM ERROR: The file was not recognized as a valid object file

Isuru Fernando

@isuruf

Want to send a PR?

Brandon Bocklund

@bocklund

I'm not sure what the correct fix for the LLVMDoubleVisitor.loads method looks like

I'm not very fluent in C++

Isuru Fernando

@isuruf

copying the string to membuffer

membuffer = s at the beginning of loads should fix it

Brandon Bocklund

@bocklund

I'll give it a shot. I'll try to set up dev environment so I can test it

Isuru Fernando

@isuruf

(Or just send a PR with a test. CI will catch it)

Btw, pickle.loads(pickle.dumps(l)) == l is always False because they don't have __eq__ implemented.

Brandon Bocklund

@bocklund

Right

I'll open a PR first (symengine/symengine#1549) with an addition to the dump/save test to verify it fails and the test is working as intended, then I'll push the fix

Isuru Fernando

@isuruf

Thanks. I cancelled tests without LLVM

Richard Otis

@richardotis

I'm trying to get a symengine development environment set up and I'm having some trouble. SymEngine builds and the test binaries pass, but when I try to build symengine.py, I get

(calphadpy3) rotis@x86_64-conda_cos6-linux-gnu ~/git/symengine.py (master) $ python setup.py install build_ext --symengine-dir=$HOME/git/symengine/build --inplace                          
running install
running build
running build_ext
SymEngine_DIR : /home/rotis/git/symengine/build/lib/cmake/symengine
SymEngine Version : 0.4.0
-- Python include path: /home/rotis/anaconda3/envs/calphadpy3/include/python3.7m
-- Python version: 3.7
-- Python install path: /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages
-- Found CYTHON: cython
CMAKE_BUILD_TYPE        : Release
CMAKE_CXX_FLAGS         : -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe 
CMAKE_CXX_FLAGS_RELEASE : -Wall -Wextra -Wno-unused-parameter -fno-common -O3 -march=native -funroll-loops -std=c++11 -fPIC -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DNDEBUG -fopenmp
CMAKE_CXX_FLAGS_DEBUG   : -Wall -Wextra -Wno-unused-parameter -fno-common -g -ggdb -std=c++11 -fPIC -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -fopenmp
HAVE_SYMENGINE_MPFR     : False
HAVE_SYMENGINE_MPC      : False
HAVE_SYMENGINE_PIRANHA  : False
HAVE_SYMENGINE_FLINT    : False
HAVE_SYMENGINE_LLVM     : True
Copying source of python wrappers into: /home/rotis/git/symengine.py
-- Configuring done
-- Generating done
-- Build files have been written to: /home/rotis/git/symengine.py
[ 25%] Building CXX object symengine/lib/CMakeFiles/symengine_wrapper.dir/symengine_wrapper.cpp.o
In file included from /home/rotis/git/symengine/build/lib/cmake/symengine/../../../include/symengine/mp_class.h:7:0,
                 from /home/rotis/git/symengine.py/symengine/lib/symengine_wrapper.cpp:620:
/home/rotis/git/symengine/build/lib/cmake/symengine/../../../include/symengine/mp_wrapper.h:5:10: fatal error: gmp.h: No such file or directory
 #include <gmp.h>
          ^~~~~~~
compilation terminated.
gmake[2]: *** [symengine/lib/CMakeFiles/symengine_wrapper.dir/build.make:69: symengine/lib/CMakeFiles/symengine_wrapper.dir/symengine_wrapper.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:177: symengine/lib/CMakeFiles/symengine_wrapper.dir/all] Error 2
gmake: *** [Makefile:130: all] Error 2
error: error building project

I am on master for both

Richard Otis

@richardotis

Also I have gmp installed on this computer, and my include file is at /usr/include/gmp.h

Richard Otis

@richardotis

It looks like adding "-I/usr/include" to CMAKE_CXX_FLAGS fixed the issue. I think it may be some kind of poor interaction between my conda environment and my system's cmake

specifically, cmake -DCMAKE_CXX_FLAGS="-I/usr/include -L/usr/lib64" seemed to do the trick

Richard Otis

@richardotis

I did eventually fix it and can run lambdify from symengine.py. The problem was some "mixing" between my conda toolchain (gcc 7) and my system toolchain (gcc 9). I rebuilt symengine completely isolated from conda. The line I used was cmake -DCMAKE_INSTALL_PREFIX=$HOME/git/symengine/build -DINTEGER_CLASS=boostmp -DWITH_LLVM=8.0 -DWITH_OPENMP=yes .

Then, still isolated from conda, I went to symengine.py and manually ran cmake -DSymEngine_DIR=$HOME/git/symengine/build . && make

Only at that point did I activate my conda environment and run pip install -e . to make symengine.py available to conda. Everything was built using my system gcc

Marcello

@torshind_gitlab

Hello there. Thanks for your awesome library.
Please have a look to this:

#include <symengine/expression.h>

int main(int argc, char* argv[]) {
  auto x = SymEngine::symbol("x");

  std::cout << "#1: " << SymEngine::Expression(SymEngine::abs(SymEngine::abs(x))) << "\n";
  std::cout << "#2: " << SymEngine::Expression(SymEngine::abs(SymEngine::mul(x, SymEngine::sign(x)))) << "\n";
  std::cout << "#3: " << SymEngine::Expression(SymEngine::floor(SymEngine::ceiling(x))) << "\n";
  std::cout << "#4: " << SymEngine::Expression(SymEngine::ceiling(SymEngine::floor(x))) << "\n";
  return 0;
}

Output:

#1: abs(abs(x))
#2: abs(x*sign(x))
#3: ceiling(x)
#4: floor(x)

Maybe I'm doing something wrong, but I'd expect #1 = #2 = abs(x)
What do you think about that?

Isuru Fernando

@isuruf

@torshind_gitlab, I think we can fix the first one, but the second one is hard. Even sympy cant do it.

Where communities thrive

People

Repo info

Activity

Reproducing the bug