Hi folks. I've been having trouble getting smile to link to blas. I'm doing a PCA on a ~
5k x 7.5k matrix and it's taking an insane amount of time (40-50min). I'm using org.clojars.haifengl/smile "2.5.0" and com.github.haifengl/smile-mkl "2.5.0"on Pop!_OS (an Ubuntu fork; version 20.04 LTS on an i9-9880H processor), using @cnuernber 's tech.ml.dataset lib. Others have subsecond times for matrices in this range, so I'm assuming I'm doing something wrong. However, my understanding from reading the smile docs is that smile-mkl should include all of the blas related binaries. I am seeing INFO smile.math.blas.BLAS - smile-mkl module is available.(and similarly for smile.math.blas.LAPACK). Can someone please help me figure out what I'm missing? Thanks!
how do you call PCA
@haifengl - is there a pure-java implementation of the new matrix API in
2.5.0? We do not want to have to depend on platform-specific natives in our application (i.e. we don't want to include BLAS or MKL dependencies).Both are pretty big (MKL is huge >120MB) and are not suitable for deployment on embedded devices. Prior to 2.5.0 we were able to deploy ML tools rather nicely and compactly using only SMILE jars
5 replies
@haifengl Thanks for confirming that openblas is included in the jars. I'm now becoming suspicious that the performance issues are upstream of smile. The
INFO smile.math.blas.BLAS - smile-mkl module is available. messages are not getting printed out until the very end of the computations. Am I correct in assuming that these messages should get printed out more or less as soon as the SVD starts? If so, that would suggest the issues are indeed upstream. Thanks again!
Was blown away when I realized you had implemented UMAP :-)
Question though: In the future might it be possible to support custom nearest-neighbor graphs, as it is with the python implementation (via igraph)?
Reason is that I'm dealing with somewhat skewed data in terms of sparseness (some rows are relatively dense, while others have only a handful of results), and a custom distance metric has proved very helpful for teasing apart the relationships in these datasets.
If that's something you'd consider, I'd be happy to file an issue for tracking.
Again, thanks for all of your work on this :-)
OK; I actually discovered UMAP by looking at the code. Now that I'm looking at the docs, it's clearer: https://haifengl.github.io/api/java/smile/manifold/UMAP.html
I see that you can specify both the adjacency graph on initialization as well as the actual data and distance to the
of method.
This seems a little odd to me as in my understanding (and at least in the python implementation) the distance metric is only (primarily?) used for computing the nearest neighbors graph. But in smile, the NN graph is passed in directly in the UMAP initialization. Please let me know if I'm misunderstanding something. Thanks!
customized
distance is to create nearest neighbor graph
Right; Those are static methods
Does that mean its not possible to specify a specific graph, vs KNN?
Thanks for explaining all of that.
It shouldn't be necessary at the moment for what I'm working on, since we're using KNN for our graphs, but would you be open to supporting UMAP on a custom
AdjacencyList/graph object?
Happy to throw up an feature request issue for tracking if this is something you'd consider.
Separately, any chance that you'd be interested in adding the Leiden clustering algorithm? https://www.nature.com/articles/s41598-019-41695-z? :-)
@haifengl Appears that the native lib jars have changed from 2.4.0 to 2.5.0. Earlier they were using netlib, now its a combination of openblas and arpack. Is there a difference in functionality or performance? I am on Linux. In the Scala REPL, sometimes there are name clashes like
reference to dot is ambiguous; it is imported twice in the same scope by import smile.data.formula._ and import smile.math.MathEx.What are the recommended interfaces for Scala users to leverage the linear algebra and math routines? I did a import smile.math.MathEx.dot in the case above. Thanks.
@haifengl I did some more digging on the PCA performance issues and was able to rule out something upstream of smile causing the slowdown. Looking at the implementation, it seems as if the only thing upstream of the SVD itself is the recentering step (and copying of data into a matrix):
double[] mu = MathEx.colMeans(data);
Matrix x = new Matrix(data);
for (int j = 0; j < n; j++) {
for (int i = 0; i < m; i++) {
x.sub(i, j, mu[j]);
}
}
Please let me know if anything comes to mind about why this might be so slow. Thanks!
@kamilkloch we don't need
smile-netlibany more. OpenBLAS/MKL is in use from 2.5.0
@haifengl Thanks, how do I now check if native BLAS/LAPACK/ARPACK libraries are loaded? With 2.2.xI would do [BLAS/LAPACK/ARPACK].getInstance().getClass.getName contains "NativeSystem". In 2.5.0 [BLAS/LAPACK].getInstance().getClass.getName returns smile.math.blas.openblas.OpenBLAS and ARPACK.getInstance() does not exist...
project dependencies:
```
libraryDependencies ++= Seq(
"com.github.haifengl" %% "smile-scala" % "2.5.1",
"org.bytedeco" % "javacpp" % "1.5.3" classifier "windows-x86_64" classifier "linux-x86_64",
"org.bytedeco" % "openblas" % "0.3.9-1.5.3" classifier "windows-x86_64" classifier "linux-x86_64",
"org.bytedeco" % "arpack-ng" % "3.7.0-1.5.3" classifier "windows-x86_64" classifier "linux-x86_64"
)
```
libraryDependencies ++= Seq(
"com.github.haifengl" %% "smile-scala" % "2.5.1",
"org.bytedeco" % "javacpp" % "1.5.3" classifier "windows-x86_64" classifier "linux-x86_64",
"org.bytedeco" % "openblas" % "0.3.9-1.5.3" classifier "windows-x86_64" classifier "linux-x86_64",
"org.bytedeco" % "arpack-ng" % "3.7.0-1.5.3" classifier "windows-x86_64" classifier "linux-x86_64"
)
*ends up