Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 03 23:52
  • Dec 03 12:58
    nkitagrawal starred symengine/symengine
  • Nov 22 13:34
    csavur starred symengine/symengine
  • Nov 21 21:28
    certik closed #1620
  • Nov 21 21:28
    certik commented #1620
  • Nov 17 23:57
    lookuptables starred symengine/symengine
  • Nov 16 11:59
    miRoox starred symengine/symengine
  • Nov 14 13:07
    jmig5776 commented #1625
  • Nov 13 20:14
    certik commented #1625
  • Nov 13 20:14

    certik on master

    Added method cancel which cance… Merge pull request #1625 from j… (compare)

  • Nov 13 20:14
    certik closed #1625
  • Nov 13 19:29
    jmig5776 commented #1625
  • Nov 13 17:01
    jmig5776 commented #1625
  • Nov 13 17:01
    jmig5776 synchronize #1625
  • Nov 13 16:48
    jmig5776 commented #1625
  • Nov 13 16:46
    certik commented #1625
  • Nov 13 16:45
    certik commented #1625
  • Nov 13 15:08
    jmig5776 reopened #1625
  • Nov 13 15:08
    jmig5776 closed #1625
  • Nov 13 14:47
Richard Otis
@richardotis
(calphadpy3) [rotis@kaufman pycalphad]$ ldd /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/symengine_wrapper.cpython-37m-x86_64-linux-gnu.so
        linux-vdso.so.1 =>  (0x00007ffc20ffa000)
        libsymengine.so.0.4 => /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../../libsymengine.so.0.4 (0x00007f1867949000)
        libflint.so.13 => /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../../libflint.so.13 (0x00007f18672fd000)
        libmpc.so.3 => /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../../libmpc.so.3 (0x00007f18672e2000)
        libmpfr.so.6 => /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../../libmpfr.so.6 (0x00007f1867263000)
        libstdc++.so.6 => /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../../libstdc++.so.6 (0x00007f1867121000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f1866df6000)
        libgcc_s.so.1 => /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../../libgcc_s.so.1 (0x00007f1866de1000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f1866a1d000)
        libgmp.so.10 => /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../.././libgmp.so.10 (0x00007f1866789000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f1866584000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1866368000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f18696ba000)
        libntl.so.33 => /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../.././libntl.so.33 (0x00007f1865f3a000)
        libgf2x.so.1 => /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../../././libgf2x.so.1 (0x00007f1865d2d000)
Isuru Fernando
@isuruf

What's the output of,

nm -g /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../../libsymengine.so.0.4 | grep _ZN9SymEngine17LLVMDoubleVisitor5loadsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

?

Richard Otis
@richardotis
(calphadpy3) [rotis@kaufman pycalphad]$ nm -g /home/rotis/anaconda3/envs/calphadpy3/lib/python3.7/site-packages/symengine/lib/../../../../libsymengine.so.0.4 | grep _ZN9SymEngine17LLVMDoubleVisitor5loadsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE00000000003e0e80 T _ZN9SymEngine17LLVMDoubleVisitor5loadsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
00000000003e0e80 T _ZN9SymEngine17LLVMDoubleVisitor5loadsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Isuru Fernando
@isuruf
Hmm, you'll have to link to libsymengine.so. I see no way around that
Richard Otis
@richardotis
Would adding it as a libraries argument in the Extension do the trick
Isuru Fernando
@isuruf
Yes, libraries = ['symengine'] should work
Richard Otis
@richardotis
Looks like that did the trick! Thanks again
Brandon Bocklund
@bocklund
Just wanted to check in again - we've transitioned all the codegen of pycalphad from SymPy+Cython to symengine. Our test suite has gone from taking almost 30 minutes on my local dev machine to 3m 30s! 30s of that is just calling sympify, which we hope to remove in the future (I think pickle support is a blocker). The codegen and differentiation are about 10x faster and it looks like calling our objective is somewhere around 2x faster. The codegen boost is really impactful for our users and startup times will go from 1-5 minutes to just a few seconds. Thank you for this effort!
Isuru Fernando
@isuruf
Nice. We would like to have serialization, but unfortunately none of us have experience in doing that in C++. symengine/symengine#1394
Brandon Bocklund
@bocklund
Yeah, we saw that. I don't have any experience either, unfortunately.

Any idea what some use cases of symengine would be where you need guarantees of compatibility as mentioned in the last comment?

Their 2 main downsides are that 1) they serialize into some custom format, which is not much usable as a data exchange format with other libraries/programs 2) they offer few guarantees of forward compatibility (e.g., if you serialize an object with Boost 1.66 it's not guaranteed to be deserializable with Boost 1.65).

Isuru Fernando
@isuruf
@bocklund, a use case is where you pickle an object and I unpickle it
Brandon Bocklund
@bocklund
IIRC pickles are guaranteed to be Python version independent, but not platform independent
Isuru Fernando
@isuruf
pickles are supposed to be backward compatible
and independent of platform
if you pickle in python 2 on osx, you can unpickle in python 3 on linux
you can't pickle in python3 on osx and unpickle in python 2 on linux
so, pickle is not forward compatible, but if 2 people used the same python minor version, they'll be compatible. If we use boost, there's an additional constraint on boost version
Richard Otis
@richardotis
One use case where you're less sensitive to environment compatibility is using dask or multiprocessing, where serialization is basically just IPC
Though I very much understand that you don't want to implement serialization with a footgun that users might not notice
Richard Otis
@richardotis
Are the Windows symengine packages on conda-forge built with LLVM enabled?
Isuru Fernando
@isuruf
Yes. And wheels too
Richard Otis
@richardotis
And it's built with MSVC?
Richard Otis
@richardotis
I am trying to link to symengine.lib using the g++ that comes with the mingw-w64 package. It finds it, but I'm getting all undefined references during the compile step.
Isuru Fernando
@isuruf
Yes, they are built with MSVC
Richard Otis
@richardotis
Okay, looks like I'll need to try converting my toolchain over. Thanks.
Isuru Fernando
@isuruf
To use with mingw-w64, you'll have to use the C interface instead of C++
Richard Otis
@richardotis
Do I have any control over that from the Cython side?
Isuru Fernando
@isuruf
Sure. You can use the functions in cwrapper.h instead of the C++ classes
Richard Otis
@richardotis
That would mean recreating and vendoring the portion of symengine_wrapper I'm using?
Isuru Fernando
@isuruf
Yes, mostly the LLVMDoubleVisitor related functions
Isuru Fernando
@isuruf
@richardotis, @bocklund I looked at the pycalphad PR. One thing I saw was that you were building lots of LLVMDoubleVisitors. Are they called with the same input? If so, you might be missing out on cse optimizations.
Richard Otis
@richardotis
They usually are, but we can't guarantee that due to the structure of our optimization problem.
Richard Otis
@richardotis
We do some grouping of our constraint functions, which does get the CSE benefit
We are SymEngine and LLVM newbies though, if there are any new superpowers we may have gained, please let us know. ;)
Isuru Fernando
@isuruf
Can you post the symbolic expressions for something like mass_hess?
Richard Otis
@richardotis
We could probably pickle the sympy versions of the objects
Isuru Fernando
@isuruf
yeah, that works
python >3.4
Richard Otis
@richardotis
@bocklund probably has a good one for a complex system. With the latest update we're hoping to push beyond the limits of what the commercial codes can do in terms of function complexity
Isuru Fernando
@isuruf
For something like mass_hess, it might be beneficial to have a loop with the data in column major format to get SIMD benefits
Brandon Bocklund
@bocklund

Pickling and unpickling the LLVMDouble does not correctly reset the LLVMDoubleVisitor that is supposed to be in llvm_double[0].

Reproducing the bug

from symengine import symbols
x, y, z = symbols('x y z')
ex = x**2 + y**2 + z**2
from symengine import lambdify
l = lambdify([x, y, z], [ex], backend='llvm')
import pickle
print('pickle.loads(pickle.dumps(l)) == l  :: ', pickle.loads(pickle.dumps(l)) == l)
print(l.__reduce__())
print(pickle.loads(pickle.dumps(l)).__reduce__())

gives

pickle.loads(pickle.dumps(l)) == l  ::  False
(<built-in function llvm_loading_func>, (3, 1, [()], True, 1, 'C', [0, 1], <class 'numpy.float64'>, b"\xcf\xfa\xed\xfe\x07\x00\x00\x01\x03\x00\x00\x00\x01\x00\x00\x00\x04\x00\x00\x00`\x01\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x19\x00\x00\x00\xe8\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x00\x80\x01\x00\x00\x00\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x00\x07\x00\x00\x00\x07\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00__text\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00__TEXT\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'\x00\x00\x00\x00\x00\x00\x00\x80\x01\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00__eh_frame\x00\x00\x00\x00\x00\x00__TEXT\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00(\x00\x00\x00\x00\x00\x00\x008\x00\x00\x00\x00\x00\x00\x00\xa8\x01\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0b\x00\x00h\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00$\x00\x00\x00\x10\x00\x00\x00\x00\x0e\n\x00\x00\x00\x00\x00\x02\x00\x00\x00\x18\x00\x00\x00\xe0\x01\x00\x00\x01\x00\x00\x00\xf0\x01\x00\x00\x10\x00\x00\x00\x0b\x00\x00\x00P\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf2\x0f\x10\x07\xf2\x0f\x10O\x08\xf2\x0f\x10W\x10\xf2\x0fY\xd2\xf2\x0fY\xc0\xf2\x0fX\xc2\xf2\x0fY\xc9\xf2\x0fX\xc8\xf2\x0f\x11\x0e\xc3\x00\x14\x00\x00\x00\x00\x00\x00\x00\x01zR\x00\x01x\x10\x01\x10\x0c\x07\x08\x90\x01\x00\x00\x1c\x00\x00\x00\x1c\x00\x00\x00\xb8\xff\xff\xff\xff\xff\xff\xff'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x0e\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00___unnamed_1\x00\x00\x00"))
(<built-in function llvm_loading_func>, (3, 1, [()], True, 1, 'C', [0, 1], <class 'numpy.float64'>, b''))

You can see that the pickled/unpickled version gives an empty byte string for the dumped LLVMDoubleVisitor bytes. @richardotis and I were tracking this down and the issue looks to be that the membuffer is not correctly reset in LLVMDoubleVisitor.loads here when called in self.llvm_double[0].loads as part of the LLVMDouble._load method here. In that case, the root fix would be to set the membuffer in LLVMDoubleVisitor.loads in the same way as LLVMDoubleVisitor.__init__.

A test for this might be to do pickle.loads(pickle.dumps(pickle.loads(pickle.dumps(l)))), which currently will kill the kernel with a LLVM ERROR: The file was not recognized as a valid object file

Isuru Fernando
@isuruf
Want to send a PR?
Brandon Bocklund
@bocklund
I'm not sure what the correct fix for the LLVMDoubleVisitor.loads method looks like
I'm not very fluent in C++
Isuru Fernando
@isuruf
copying the string to membuffer
membuffer = s at the beginning of loads should fix it
Brandon Bocklund
@bocklund
I'll give it a shot. I'll try to set up dev environment so I can test it
Isuru Fernando
@isuruf
(Or just send a PR with a test. CI will catch it)