fergiemcdowall/search-index

A persistent full text search engine for the browser and Node.js

Fergus McDowall

@fergiemcdowall

Would be really interested in getting a look at your setup

Jonathan Poltak Samosir

@poltak

@fergiemcdowall Not sure exactly what would be of interest, so here's a gist (non-runnable) containing an example doc that gets indexed, simple query with a filter, a more realistic (in relation to use within the program) query with a filter, an example scoringCriteria object from a search result with a NaN score, plus the index/field options being used:
https://gist.github.com/poltak/6c8c003ea8a59407be5b929e2002e418

Bit of a mess, so let me know if you need clarification on any parts or the reasonings for certain choices (and no rush looking at this; everyone has stuff to do). As I was putting it together, line 89 in the scoringCriteria dump stood out to me - not sure if there should be a null value there.

Fergus McDowall

@fergiemcdowall

In order to fix an indexing/search bug the project needs:

1) a simple script that reproduces the error by indexing one or more documents and then running a query (dont have to bother with a test framework- just dump to console.log)
2) a note on the expected vs actual result.

a very specific error report: "indexing documents containing the word 'avacado' crashes my system"

But basically- if you provide a runnable script, it will always get fixed

Jonathan Poltak Samosir

@poltak

Totally understood. Sadly I don't have much time to do anything lately, but will try to get a runnable script to you in the next few weeks if I can't resolve it trying other things in the meantime. Got some ideas on ways I could do the queries differently to see how the results differ

Jonathan Poltak Samosir

@poltak

@fergiemcdowall after encountering a much more severe issue regarding unexpected results (using offset + pageSize pagination query opts) with our index implementation, and being completely lost with it, I wrote up a node script this morning to try and reproduce it. Sadly I haven't been able to reproduce this one in node (yet). As I already had the script working I instead decided to try and get it to reproduce those NaN scores on filter search I was talking about last week.

While in a lot of queries I (frustratingly) cannot get it to happen as it is happening in our browser implementation, however I did found a query that manages to return all the expected results but with NaN scores.
Here's the gist that should be runnable in >=Node 7.6:
https://gist.github.com/poltak/7eb8128d9bbc6d784a38a922f8d8560c

No rush having a look. Some simple instructions up there, but let me know if you need any clarification on what or why I'm doing things - I expect it's something I'm doing wrong. The data is taken directly from our app, although note that I'm cutting out a lot of the content field as it's not really important for the purposes here (and can be quite long). The index opts are as our index is set up.

Fergus McDowall

@fergiemcdowall

great- will take a look

have opened an issue here: fergiemcdowall/search-index#413

Jonathan Poltak Samosir

@poltak

Thanks @fergiemcdowall. Appreciate it.

swissums

@swissums

I would like to see the index-time of individual words as they are indexed. I was thinking just a simple console.time(), Where is the best place to add that into search-index?

Espen Klem

@eklem

@swissums Somewhere in search-index-adder ?

Espen Klem

@eklem

I don't know the add-to-index part well enough, but it's a place to start looking.

Fergus McDowall

@fergiemcdowall

Thats a good question @swissums

There's a few ways to do that

maybe in here? https://github.com/fergiemcdowall/search-index-adder/blob/master/lib/add.js

swissums

@swissums

@fergiemcdowall Yeah I managed to fit it in there quite nicely thanks for the help =)

Espen Klem

@eklem

:+1:

Espen Klem

@eklem

@fergiemcdowall is the /dist folder the browserified version of search-index?

Fergus McDowall

@fergiemcdowall

@eklem yes!

Espen Klem

@eklem

Thank you =) I'll start making norch-vue-app ready for truly running in the browser.

Espen Klem

@eklem

I think I've done the same error with norch-vue-app as you did with norch-angular-app, making it too slick/too much overhead. I'll try to make a simpler integration between norch/search-index and Vuejs. What I need is support for all the endpoints/functions of norch/search-index

Fergus McDowall

@fergiemcdowall

@eklem easily done! :)

Espen Klem

@eklem

Yes, wanted to do two tasks in one go. Make a framework everybody could use AND make it look good.

Ducros Alix

@Karalix

Hi, I have a question about the difference between add() and concurrentAdd() method : in most examples, before using add(), si.defaultPipeline(batchOptions) is called. If I want to use concurrentAdd() instead of add(), do I still need to do a similar call ? How could I if I have not a stream ?

Working on 0.9.16 if it can be of any help

Fergus McDowall

@fergiemcdowall

Hi Ducros! Yes, you have to use a pipeline- it turns the data into a digestible format

.concurrentAdd may work without

There are some examples in the tests

Ducros Alix

@Karalix

Thanks :smile:

Mustafa Hanif

@mustafa-hanif

how to load index from file

Espen Klem

@eklem

Have you exported an index, @mustafa-hanif? And is it norch or search-index you're working with?

Espen Klem

@eklem

I think we need to understand a little what you are trying to achieve and how. Any code you can show us, or an explanation of your setup?

Espen Klem

@eklem

From GitHub: request for comment - move to promises-based API? fergiemcdowall/search-index#446

Espen Klem

@eklem

How is the move going, @fergiemcdowall ?

Fergus McDowall

@fergiemcdowall

Making progress. Doing it in a private repo

Vladimir Vujosevic

@frontmesh

Hi everyone, I have a question regarding search-index usage. I have a large collection in MongoDB with over 2 million documents. Performing a regex query can take up from 1s to over 10s in some cases. Is it possible to improve speed of these queries by using search-index? is this a use case for this library at all ?

Espen Klem

@eklem

Yes. I think it was tested with 1 or 5 million documents, but got rewritten to have a streaming interface to work better with large document sets. I'm not sure what will be the actual response time, but I'm guessing it's quite a lot better.

Vladimir Vujosevic

@frontmesh

Thanks @eklem. Is there maybe any example on how to connect it with mongodb collection and read/query/search data from it ?

Fergus McDowall

@fergiemcdowall

@frontmesh no documentation around that Im afraid

Vladimir Vujosevic

@frontmesh

@fergiemcdowall Thanks. I'll see if i can come up with something

Espen Klem

@eklem

Just a heads up @all: There's a big rewrite of search-index, and breaking change, as of v1.0.0. New Promise API and you can chain and/or/not in a much more flexible way.

Espen Klem

@eklem

Browser example to come soon

prog20901

@prog20901

I am very happy to come across this group

Looking for a similar app or portable solution like below - type-ahead search with easy navigation, number of hits - ex: http://table.branham.org/#/main - where huge pre-defined text has been dumped for easy query and navigation

Is it possible via search-index? can someone help me with a demo sample or good example to build like the above

everything is very quick and fast and search results in less than a second

Fergus McDowall

@fergiemcdowall

You can achieve this with DICTIONARY. There is a bit of documentation here -> https://github.com/fergiemcdowall/search-index/blob/master/docs/API.md#how-do-i-make-a-simple-typeahead--autosuggest--matcher

prog20901

@prog20901

@fergiemcdowall Can you please help me with some UI examples so that will see how i can bring up like above? for me it looks complex. any demo or sample start would be helpful

Where communities thrive

People

Repo info

Activity