These are chat archives for bluescarni/piranha

8th
Feb 2016
Francesco Biscani
@bluescarni
Feb 08 2016 00:00
it looks like in c:\bzip2 we have static libs for bz2, but for some reason it still looks for the dynamic library instead
I wonder if we should link to the libbz2-static.lib one?
I vaguely remember random facts about libraries on windows, such as that there are 3 files per each library to be provided and that one of these contains a list of the symbols of the library
and then the dll gets generated from this descriptor or something like that
Francesco Biscani
@bluescarni
Feb 08 2016 00:15
Ok it looks like it takes too long now, the build timeouts around 70% into the compilation
I am now trying to see if we can do a parallel build
I guess alternatively we could split the test in two parts, I think you mentioned that sympy does this
Francesco Biscani
@bluescarni
Feb 08 2016 00:36
off to bed now... today there was a lot of progress! thanks again for all your work @isuruf
Isuru Fernando
@isuruf
Feb 08 2016 01:13
What sympy does is, they have 4 jobs in the build to do the testing.
Similarly what we can do in piranha is to divide the tests into 3 parts
Isuru Fernando
@isuruf
Feb 08 2016 01:32
Something like this isuruf/piranha@1ca1ebc
Francesco Biscani
@bluescarni
Feb 08 2016 08:27
@isuruf awesome! Need to drive to work, I'll try it in a couple of hours :)
Francesco Biscani
@bluescarni
Feb 08 2016 10:03
@isuruf I wanted to avoid the test_split variable to appear in the GUI... can I change SET(PIRANHA_TEST_SPLIT no CACHE Bool "Split compilation of piranha test cases for CI") to SET(PIRANHA_TEST_SPLIT no CACHE Internal "Split compilation of piranha test cases for CI")?
Francesco Biscani
@bluescarni
Feb 08 2016 11:38
@isuruf I am not exactly sure what is going on here: https://ci.appveyor.com/project/bluescarni/piranha/build/20/job/dfpsolw4y2xxrpss
so basically one of the tests fails, but when I log into the VM I cannot really run any of the tests manually from the command line
they all complain about missing dlls
gmp, bz2, etc
even if they are supposed to be statically linked in
I did set the PATH before attempting to run the tests from the command line, but still doesn't work
Isuru Fernando
@isuruf
Feb 08 2016 11:39
you have to add C:\mingw64\bin to PATH
Francesco Biscani
@bluescarni
Feb 08 2016 11:40
I wonder how CMake is managing to run the other test
I am pretty sure I did... but why do we need this if libraries are static?
ah wait ok one path was missing :)
Isuru Fernando
@isuruf
Feb 08 2016 11:41
I have no idea. How static libraries and dynamic libraries work in Windows is beyond me
Francesco Biscani
@bluescarni
Feb 08 2016 11:41
yeah it's a bit strange wrt linux
but now I can investigate the failure, thanks for the help
Isuru Fernando
@isuruf
Feb 08 2016 11:44
The way I split the test is not optimal I guess. Maybe we should have similar tests in one run like series_01, series_02 etc
Francesco Biscani
@bluescarni
Feb 08 2016 11:45
it works ok, I like it because it's automatic
Isuru Fernando
@isuruf
Feb 08 2016 11:48
It's automatic, but it takes 2.5 hours. (including dependency setup time of 6.5 minutes) against 1 hour + few minutes for just one run
Francesco Biscani
@bluescarni
Feb 08 2016 11:50
not sure I follow... how would grouping tests differently result in lower overall time?
Isuru Fernando
@isuruf
Feb 08 2016 11:50
does gcc precompile headers?
Francesco Biscani
@bluescarni
Feb 08 2016 11:50
ah I see what you mean now :)
It can, but support for it is not builtin into cmake and I am not completely sure it would help with heavily templated code
but I may be wrong
I mean if we managed to make it work reliably it would be good
but I have not much experience with precompiled headers
Isuru Fernando
@isuruf
Feb 08 2016 11:55
Maybe it was a glitch in the VM that it took longer time when I split the test
Francesco Biscani
@bluescarni
Feb 08 2016 11:58
I mean now the overall procedure does take long
since I split the big tests into small ones, the compilation time has increased
long -> longer
Isuru Fernando
@isuruf
Feb 08 2016 12:00
That is expected since the compiler now compiles the same thing more times
Francesco Biscani
@bluescarni
Feb 08 2016 12:00
yes.. it's not exactly the same thing that it generates in the end, but they are the same templates instantiated with different types
so lots of re-parsing of the same stuff
Francesco Biscani
@bluescarni
Feb 08 2016 12:50
it's really hard to debug with the time limit on the VM
I think I'll have to setup some kind of local environment
Francesco Biscani
@bluescarni
Feb 08 2016 12:56
damn I hoped we could merge this today... oh well :)
@isuruf when the time comes, is it ok if I merge from my branch into master or would you prefer me to send you a PR?
Isuru Fernando
@isuruf
Feb 08 2016 13:01
That's totally fine with me
I mean merging from your branch into master
Francesco Biscani
@bluescarni
Feb 08 2016 13:02
ok... I'll merge back from yours if you do any additional modification
Francesco Biscani
@bluescarni
Feb 08 2016 13:48
https://ci.appveyor.com/project/bluescarni/piranha so it's not too bad, there's another failure related to bzip2/filenames stuff, but at least the second batch of tests compiles and runs cleanly
each run averages around 25/30 minutes
I'll try tonight to set the toolchain up on my windows installation and investigate the two issues
the only troublesome one is the first issue, it's an assertion failure which unnerves me
Isuru Fernando
@isuruf
Feb 08 2016 13:54
are you connected to the VM at the moment?
Francesco Biscani
@bluescarni
Feb 08 2016 13:57
I was but it timed out
I managed to isolate the portion of code where the problem happens at least
I would not be surprised if it was some compiler bug
an assertion triggers in a destructor on an object which should not be destroyed
Francesco Biscani
@bluescarni
Feb 08 2016 14:02
it might be passed as a copy instead of perfectly forwarded, that would explain the extra dtor call
anyway I'll investigate tonight
Hopefully once all the issues are ironed out it should be smoother sailing :) I think as long as we keep the test cases reasonably short we should not have too many more issues in the future
Isuru Fernando
@isuruf
Feb 08 2016 14:07
Yes and we'll have to deal with only one problem at a time
Francesco Biscani
@bluescarni
Feb 08 2016 14:07
yeah CI looks like one of those things that it's better started early... like thesis writing :)
Isuru Fernando
@isuruf
Feb 08 2016 14:08
:D
You can also split the tests in Travis-CI which can run 5 parallel jobs at a time, while appveyor runs the jobs serially
Francesco Biscani
@bluescarni
Feb 08 2016 14:09
I thought about that, but on the other hand if it manages to complete in one run then we can use the other slots for other builds righT?
I was thinking of re-adding Debug GCC build now that the tests use less memory
Isuru Fernando
@isuruf
Feb 08 2016 14:10
Ah yes. You can also test OS X
Francesco Biscani
@bluescarni
Feb 08 2016 14:11
yep that would be good as well.. they have a clang VM or still GCC 4.2?
in any case it's all stuff we should worry about as a second iteration
Isuru Fernando
@isuruf
Feb 08 2016 14:11
In SymEngine, we test AppleClang and homebrew installed gcc
Francesco Biscani
@bluescarni
Feb 08 2016 14:12
only possibility for Piranha is AppleClang I guess
ah wait
homebrew
so it's 4.8 or later
Isuru Fernando
@isuruf
Feb 08 2016 14:13
yes. 4.8.3
Francesco Biscani
@bluescarni
Feb 08 2016 14:13
ok that should be good...
osx has other issues that would need to be sorted out, but at least the current tests should work
Isuru Fernando
@isuruf
Feb 08 2016 14:14
That should be left to another time, as how to install dependencies might take some time to figure out
Francesco Biscani
@bluescarni
Feb 08 2016 14:14
I had some code for thread affinity for OSX but unfortunately I never merged it and lost it when I switched jobs :/
yes right now I'd like just to get appveyor up and running
there's also python bindings testing which is another can of worms
:)
Isuru Fernando
@isuruf
Feb 08 2016 14:15
yes, as I remember correctly python bindings take lots of memory
Francesco Biscani
@bluescarni
Feb 08 2016 14:15
yes but I split them, they should take < 4G now
I would expect clang on travis to be fine, GCC might need some flag to curb the size or reduce debug info or something like that.. we'll see
https://github.com/wjakob/pybind11 eventually I am planning to switch from Boost Python to this for the bindings
I am experimenting with it in another project and it's really promising
but that'll happen in the future
Francesco Biscani
@bluescarni
Feb 08 2016 14:20
medium/longish term
Francesco Biscani
@bluescarni
Feb 08 2016 18:35
bah
cannot reproduce the first issue on my local machine
I am using the same setup as a the appveyor YAML script, including mingw, gmp, mpfr, bzip2
the only difference is that I am on windows 7
Francesco Biscani
@bluescarni
Feb 08 2016 18:51
mrmrm