fergiemcdowall/search-index

A persistent full text search engine for the browser and Node.js

Fergus McDowall

@fergiemcdowall

(due to the non-ascii ø)

Jeff See

@jeffsee55

Ok, so if I had an item like:

  const data = [
    {
      name: 'Homer Simpson',
      comments: ['doh', 'this is another comment'],
      _id: 'content/authors/homer.md'
    }
  ]

Fergus McDowall

@fergiemcdowall

therefore its easier to do ['bøker', 'er', 'gøy']

You could also introduce ngrams: ['bøker er', 'er gøy']

Jeff See

@jeffsee55

Would I get a hit for:

  await db
    .QUERY({
      FIELD: ['comments'],
      VALUE: 'another'
    })
    .then(print

Or does that need to be indexed manually by me

Fergus McDowall

@fergiemcdowall

ëtc.

lets see...

Jeff See

@jeffsee55

Ok that's what I'm seeing too I just wasn't sure if I was doing something wrong

Fergus McDowall

@fergiemcdowall

 const data = [
    {
      name: 'Homer Simpson',
      comments: 'doh this is another comment',
      _id: 'content/authors/homer.md'
    }
  ]

This would tokenize comments to 'doh', 'this', 'is', 'another', 'comment'

and you would then get a hit for VALUE: 'another'

comments: ['doh', 'this is another comment'] would allow you to search for VALUE: 'doh' and VALUE: 'this is another comment'

Jeff See

@jeffsee55

Ok, thanks for the explanation. I'm trying to see if I can use this for full search of potentially complex objects. Some might have an array of objects - which I'd like to sort of "flatten" out into something that gets indexed in the normal way

But maybe what I'm starting to understand is that there's no reason to keep those values as arrays, I can just merge them as you've done in the last example into a single string

Fergus McDowall

@fergiemcdowall

Yes, for american english, that is probably the easiest

If you want

to preserve spesial chars and punctuation, then arrays are the way to go

Jeff See

@jeffsee55

to preserve spesial chars and punctuation, then arrays are the way to go

But using arrays will also remove the ability to search for partial matches, correct? And to support those I'd have to use ngrams

Fergus McDowall

@fergiemcdowall

yes, you could also look into DICTIONARY

Nice to talk to you Jeff! I need to pick up my kid from kindergarten now, but I can get back to you later if you have any more questions

Jeff See

@jeffsee55

Thanks @fergiemcdowall for the help, I'll see if I can learn more about DICTIONARY

brightinnovator

@brightinnovator

I have a question on javascript. can someone help me on the javascript issue?

brightinnovator

@brightinnovator

I want to read 3 crore csv rows which is of 2GB csv file size and need to insert into MySQL via Java.

Could someone please help me know the fastest and memory efficient way to avoid out of memory exception as well load in lesser time?

Please kindly advise.

goodev2021

@goodev2021

how to develop an full content text search application using search-index with watermelon db (https://watermelondb.now.sh/)?

Fergus McDowall

@fergiemcdowall

You would need to find a leveldown that works for watermelondb

Where communities thrive

People

Repo info

Activity