Hi Alex, I've just started playing with ElastikNN and have a question about the score returned by elasticsearch. I'm using Jaccard LSH and seeing scores around 0.05 or less, even for the first result which should be an exact match. Am I misunderstanding how the score should work?
Fixed that bug here: alexklibisz/elastiknn#134
I'm gonna cut a release with some changes I made over the weekend and then include this in the next release
I'm gonna cut a release with some changes I made over the weekend and then include this in the next release
@bishopaj_gitlab Fix for unsorted vectors released here: https://github.com/alexklibisz/elastiknn/releases/tag/0.1.0-PRE31
Let me know if you have any other questions
Let me know if you have any other questions
I'm only just starting to look at the code, but does this also apply to the document vector? I mean when I store documents with a sparse bool vector do the entries need to be sorted?
Also a somewhat unrelated question: what are the implications of increasing the dimensionality of the document/query vectors? I'm currently using 25,000 which I took from one of your examples, but might want to increase this to reduce collisions (I'm hashing terms into these 25k buckets).
@bishopaj_gitlab The plugin will sort
trueIndices both when indexing new vectors and when running queries. So you don't need to worry about it at all. If you are curious, the reason for this is that it's much faster to compute the intersection of two sorted arrays of indices than it is to compute the intersection of unsorted arrays.
The implications of increased dimensionality are tricky with sparse vectors. What really has an effect is the number of true indices. Generally, having 2n true indices is going to mean 2x slower operations than having n true indices.
New release with a minor internal fix: https://github.com/alexklibisz/elastiknn/releases/tag/0.1.0-PRE32
@divdasani Hi Div. I haven't had the time yet to support multiple versions of ES. There's a ticket open for it on Github. It would involve modifying the gradle and CI setup. Nothing conceptually too challenging, just a bit tedious and you are the first to ask :). There was also a regression in the version of Lucene used beyond 7.6.2 that affected some queries. I'm not sure if they've resolved that or not.
My main focus right now is speeding up approximate queries to make a decent submission to the ann-benchmarks benchmark project. After that, I would like to go ahead and support two or three versions of ES concurrently. If you are comfortable with Gradle and github workflows (or know someone who is), I'd be open to collaborating on it, reviewing a PR, etc.
I check this board about once a week, so feel free to email me aklibisz@gmail.com if you want to chat about it.
Hi, I'm also interested in Elastiknn for a production use case, but we are currently using v7.9.2. I am trying to decide between a) downgrading to 7.6.2 to be consistent with elastiknn; b) trying to build a version of the plug-in from source that supports 7.9.2, or c) begging you to release a plug-in for 7.9.2. Any recommendations?
(already emailed w/ Eric about this, just posting for anyone else who might be interested)
@ejackson-eb @divdasani I got it to compile with 7.9.2 yesterday. But there are some runtime errors. The short story is that between 7.8.x and 7.9.x they did some internal refactoring to the code that lets you define custom data types. There were a handful of constructor params I had to fill in to extend the necessary classes and I bet I messed some of them up on the first pass. If anyone wants to tinker, there is a WIP PR on the project. I'll hopefully pick back up on it over the weekend.
Hey folks, here is a build of Elastiknn on ES 7.9.2 which passes all of the tests: https://github.com/alexklibisz/elastiknn/releases/tag/0.1.0-PRE42-PR173-SNAPSHOT
regarding the approximate query with LSH, does it work if the vectors are stored in nested fields and what kind of performance impact would there be?
i'm assuming that storing the sentence vectors as nested fields of each document would still allow them to be queried for vector similarity using Elastiknn, and I also want to leverage on the LSH function so that not all sentences in each document returned by keyword search need to be scored, only the nearest sentences in each document