These are chat archives for elemental/chat

18th
Feb 2017
Aidan Dang
@AidanGG
Feb 18 2017 04:23
Calling LAPACK (zgesdd) on my El::Matrix through ctrl.useLAPACK doesn't give me any problems. I'll build a debug again to check that sigmaEst thing again.
Jack Poulson
@poulson
Feb 18 2017 04:59
@AidanGG You may want to try flipping the El::FlipOrClip variable, which is the main difference between El and LAPACK's D&C
Elemental defaults to clipping negative values which should be non-negative (but weren't due to rounding error) up to zero, but LAPACK's D&C negates the value.
Aidan Dang
@AidanGG
Feb 18 2017 05:00
Ah, ok. I'll give it a go.
Jack Poulson
@poulson
Feb 18 2017 05:01
both are heuristics
it's also entirely possible that they each break on different types of matrices
IMO, the "clipping" approach seems more principled, but both introduce perturbation within a factor of two of each other
also, would you mind sending me a copy of such a matrix?
I can do a deep dive to debug it
Aidan Dang
@AidanGG
Feb 18 2017 05:02
Sure, I'll upload it to my drive in a bit. It's just 128x128
Jack Poulson
@poulson
Feb 18 2017 05:03
even better!
Jack Poulson
@poulson
Feb 18 2017 05:06
is this again from a 64-bit build?
Aidan Dang
@AidanGG
Feb 18 2017 05:06
Complex<double> 64 bit ints
I get the issue on the 2x2 process grid.
Jack Poulson
@poulson
Feb 18 2017 05:13
what about with 1x1?
or 1x2?
Aidan Dang
@AidanGG
Feb 18 2017 05:14
both of those seem to be ok
bidiagSVDCtrl.useQR also seems to be ok
Jack Poulson
@poulson
Feb 18 2017 05:16
OK, I will get to the bottom of it
I apparently need to rebuild El before I can do so though
Aidan Dang
@AidanGG
Feb 18 2017 05:16
No worries, there's no rush. Thanks for your help and expertise.
FLIP_NEGATIVES doesn't seem to help.
Jack Poulson
@poulson
Feb 18 2017 05:25
no problem; test matrices are invaluable
Jack Poulson
@poulson
Feb 18 2017 06:10
what compiler and flags are you using?
Aidan Dang
@AidanGG
Feb 18 2017 06:11
gcc 6.3.1 with OpenBLAS 0.2.19 64bit ints
Jack Poulson
@poulson
Feb 18 2017 06:11
You can check the flags in the CXX_FLAGS variable in CMakeCache.txt
I don't see any error for any grid configuration
though I have a Debug build: it's possible the compiler did an overly aggressive optimization
I will try with a Release build to see if there is a change
the secular equation solvers are particularly sensitive to associativity assumptions
Aidan Dang
@AidanGG
Feb 18 2017 06:14
I'm just using the default -O3 -fcx-fortran-rules for the release build
Jack Poulson
@poulson
Feb 18 2017 06:15
compilers are unfortunately not perfect
I'm not concluding that this is the problem yet, but it was previously a known issue with the intel compilers
Aidan Dang
@AidanGG
Feb 18 2017 06:16
I'll also try a debug build on my end
Jack Poulson
@poulson
Feb 18 2017 06:16
I'm compiling a 64-bit-int Release build and should know in 15 minutes or so
thanks!
Aidan Dang
@AidanGG
Feb 18 2017 06:29
Ok, I'm not seeing the issue on Debug, so I think you're right.
Jack Poulson
@poulson
Feb 18 2017 06:30
can you try a release build with -O3?
err, -O2?
I'm still only at 29% on my Release build
Aidan Dang
@AidanGG
Feb 18 2017 06:30
would that just involve me changing the ElVars file that I use in my makefile? I don't have to rebuild do I?
Jack Poulson
@poulson
Feb 18 2017 06:31
you would have to do a fresh build :-(
if I can reproduce the problem here I can take over
Aidan Dang
@AidanGG
Feb 18 2017 06:32
I'll get started on a build. It doesn't take too long for me with make -j4
Jack Poulson
@poulson
Feb 18 2017 06:44
also, it's possible to introduce #pragma statements into the code to try to avoid unsafe math optimizations
Aidan Dang
@AidanGG
Feb 18 2017 06:44
-O2 on Release seems to work.
Jack Poulson
@poulson
Feb 18 2017 06:45
then it seems -O3 is indeed doing some overly aggressive floating-point optimizations
it would be a shame to need to use -O2 for every file
Jack Poulson
@poulson
Feb 18 2017 07:11
I see the issue in Debug and Release builds with 2 MPI processes
Aidan Dang
@AidanGG
Feb 18 2017 07:19
Would it be worth going to -Os or -O1 or -O0?
Jack Poulson
@poulson
Feb 18 2017 07:19
I don't think it is the optimization levels anymore
I will look into it
Jack Poulson
@poulson
Feb 18 2017 07:25
Getting the 31'st singular value of the following secular SVD problem seems to be failing:
whichValue=31
d
0 
1.058049026775339e-14 
3.264165070167235e-14 
5.678278474169334e-14 
7.549675375178582e-14 
8.999682272132982e-14 
1.221972146245e-13 
1.31165743725131e-13 
1.509690856648426e-13 
1.935346943178186e-13 
2.227587276398722e-13 
2.484596565403951e-13 
2.735452664152349e-13 
2.95093712074214e-13 
3.072195747847108e-13 
3.312626545827337e-13 
3.576227579361058e-13 
4.11481521016379e-13 
4.368380737442849e-13 
4.654660121458798e-13 
4.917724349672351e-13 
5.351048585795176e-13 
5.808559195909003e-13 
6.026777472426812e-13 
6.250594383003799e-13 
6.766753212719467e-13 
7.244153585408541e-13 
7.776124345286146e-13 
8.164772181710468e-13 
8.467548733006103e-13 
9.071477145232044e-13 
1.01805321890874e-12 
0.5007056599251847 
0.5083143333578168 
0.5140067545248999 
0.5300828281216288 
0.5432158063056066 
0.558449173485537 
0.5693619673962667 
0.5858774959661275 
0.6000038931533331 
0.607556229381327 
0.6195702898722641 
0.627471185160503 

rho=5.44831e-24
z
0.8825077510160421 
-0.07358221320511406 
-0.03174202316169126 
0.04131276385906558 
-0.04434525804425347 
0.06731278591028068 
-0.05539554252257473 
-0.03512118498070101 
0.04512260091902331 
-0.1180721177417332 
0.08284806075424916 
-0.07444791552043822 
0.02259784498068728 
-0.0331198865827843 
0.01047460095220662 
-0.06333080808951928 
0.08922022110821659 
-0.07376550709419979 
0.05879641325099103 
-0.08369171424713648 
0.0687436019461881 
-0.03520100042130667 
0.1502895817945899 
-0.02136545584620814 
0.04613737435858611 
-0.1283945980959928 
0.06240166369683361 
-0.02550239013481107 
0.1565066351625207 
-0.08300592774083522 
0.2288923560918465 
0.07921931672967193 
0.001204468554921371 
0.03546379899021058 
0.01490605201498635 
0.006952012394837491 
0.00430796577937792 
0.006494679505778383 
0.0121694830195294 
0.003459607121991713 
0.008990606149639263 
0.01465219982739229 
-0.003086397925868282 
0.001933556220246539
I need to get some sleep but can look into this tomorrow.
Aidan Dang
@AidanGG
Feb 18 2017 07:31
Sure thing. Thanks a lot.
Jack Poulson
@poulson
Feb 18 2017 16:28
one thing to notice is that this is a problem diag(d)^2 + rho z z^T, where || diag(d)^2 ||_2 ~= 0.36 but || rho z z^T ||_2 = | rho | ~= 10^-24.
so the rank-one update has a norm below machine precision
Jack Poulson
@poulson
Feb 18 2017 16:46
I am testing a set of changes to El::SecularSVD, El::SecularSingularValue, El::SecularEVD, and El::SecularEigenvalue that ignore the rank-one update if its relative norm is less than 10 * eps
Jack Poulson
@poulson
Feb 18 2017 18:24
nevermind, that matrix is slightly above the reasonable tolerance since one should compute the square-root of that ratio, i.e., ~10^-12
Jack Poulson
@poulson
Feb 18 2017 20:08
I think I fixed the problem
The trivial helper routine secular_svd::RelativeEigenvalueToRelativeSingularValue was to blame, as it was assuming that one of the intermediate calculations was always positive
this would be true in exact arithmetic but seems to fail in some extreme cases and I am patching it to clip to zero
Jack Poulson
@poulson
Feb 18 2017 20:13
@AidanGG I committed fixes, albeit in a bit of a rush
Ryan H. Lewis
@rhl-
Feb 18 2017 20:21
@poulson does: lapack::BidiagSVDQRAlg modify the data in the bidiagonal matrix ?
nevermind , I see it does.