fergiemcdowall/search-index

A persistent full text search engine for the browser and Node.js

Fergus McDowall

@fergiemcdowall

;(async () => {
  const si = require('search-index')
  const print = txt => console.log(JSON.stringify(txt, null, 2))
  const db = await si({
    name: 'arrays'
  })

  const data = [
    {
      name: 'Homer Simpson',
      favoritePosts: ['content/posts/welcome3.md'],
      _id: 'content/authors/homer.md'
    }
  ]

  await db.PUT(data)

  await db
    .QUERY({
      FIELD: ['favoriteposts'],
      VALUE: 'content/posts/welcome3.md'
    })
    .then(print)
})()

just to explain ->

when you use arrays, special characters are not stripped

Jeff See

@jeffsee55

Ok, makes sense why that works for me then, what's the reason for the difference?

Fergus McDowall

@fergiemcdowall

so ['content/posts/welcome3.md'] is stored in the index as 'content/posts/welcome3.md'

whereas if you simply do 'content/posts/welcome3.md' (no array), then special chars will be be stripped and the string will be tokenized

Jeff See

@jeffsee55

Would

  const data = [
    {
      name: 'Homer Simpson',
      favoritePosts: ['content/posts/welcome3.md', 'content/post/welcome4.md'],
      _id: 'content/authors/homer.md'
    }
  ]

Be indexed as 'content/posts/welcome3.md' and 'content/posts/welcome4.md' as separate fields somehow (sorry, might be a confusing question)

Fergus McDowall

@fergiemcdowall

yes

If you want to investigate this you can use level-out to inspect the index

level-out arrays
{"key":"favoriteposts:content/posts/welcome3.md#1.00","value":["content/authors/homer.md"]}
{"key":"name:homer#1.00","value":["content/authors/homer.md"]}
{"key":"name:simpson#1.00","value":["content/authors/homer.md"]}
{"key":"￮DOCUMENT_COUNT￮","value":1}
{"key":"￮DOC_RAW￮content/authors/homer.md￮","value":{"name":"Homer Simpson","favoritePosts":["content/posts/welcome3.md"],"_id":"content/authors/homer.md"}}
{"key":"￮FIELD￮favoriteposts￮","value":"favoriteposts"}
{"key":"￮FIELD￮name￮","value":"name"}
{"key":"￮￮CREATED","value":1622729161588}

Jeff See

@jeffsee55

Ah that would be awesome, I was looking for a way to to do that

Fergus McDowall

@fergiemcdowall

level-out is really handy for seeing how things are being indexed

BTW- indexing and query pipelines are going to be much better in search-index@3 (coming soon)

Jeff See

@jeffsee55

I saw that! Seems like it's close, is the reason array values don't get stripped of special characters for something specific?

Fergus McDowall

@fergiemcdowall

Yes- its so that you can easily do your own tokenization and use non-ascii chars

So for instance- I work a lot with scandinavian languages and sentences like "bøker er gøy" are not always tokenized correctly

(due to the non-ascii ø)

Jeff See

@jeffsee55

Ok, so if I had an item like:

  const data = [
    {
      name: 'Homer Simpson',
      comments: ['doh', 'this is another comment'],
      _id: 'content/authors/homer.md'
    }
  ]

Fergus McDowall

@fergiemcdowall

therefore its easier to do ['bøker', 'er', 'gøy']

You could also introduce ngrams: ['bøker er', 'er gøy']

Jeff See

@jeffsee55

Would I get a hit for:

  await db
    .QUERY({
      FIELD: ['comments'],
      VALUE: 'another'
    })
    .then(print

Or does that need to be indexed manually by me

Fergus McDowall

@fergiemcdowall

ëtc.

lets see...

Jeff See

@jeffsee55

Ok that's what I'm seeing too I just wasn't sure if I was doing something wrong

Fergus McDowall

@fergiemcdowall

 const data = [
    {
      name: 'Homer Simpson',
      comments: 'doh this is another comment',
      _id: 'content/authors/homer.md'
    }
  ]

This would tokenize comments to 'doh', 'this', 'is', 'another', 'comment'

and you would then get a hit for VALUE: 'another'

comments: ['doh', 'this is another comment'] would allow you to search for VALUE: 'doh' and VALUE: 'this is another comment'

Jeff See

@jeffsee55

Ok, thanks for the explanation. I'm trying to see if I can use this for full search of potentially complex objects. Some might have an array of objects - which I'd like to sort of "flatten" out into something that gets indexed in the normal way

But maybe what I'm starting to understand is that there's no reason to keep those values as arrays, I can just merge them as you've done in the last example into a single string

Fergus McDowall

@fergiemcdowall

Yes, for american english, that is probably the easiest

If you want

to preserve spesial chars and punctuation, then arrays are the way to go

Jeff See

@jeffsee55

to preserve spesial chars and punctuation, then arrays are the way to go

But using arrays will also remove the ability to search for partial matches, correct? And to support those I'd have to use ngrams

Fergus McDowall

@fergiemcdowall

yes, you could also look into DICTIONARY

Nice to talk to you Jeff! I need to pick up my kid from kindergarten now, but I can get back to you later if you have any more questions

Jeff See

@jeffsee55

Thanks @fergiemcdowall for the help, I'll see if I can learn more about DICTIONARY

brightinnovator

@brightinnovator

I have a question on javascript. can someone help me on the javascript issue?

brightinnovator

@brightinnovator

I want to read 3 crore csv rows which is of 2GB csv file size and need to insert into MySQL via Java.

Could someone please help me know the fastest and memory efficient way to avoid out of memory exception as well load in lesser time?

Please kindly advise.

goodev2021

@goodev2021

how to develop an full content text search application using search-index with watermelon db (https://watermelondb.now.sh/)?

Fergus McDowall

@fergiemcdowall

You would need to find a leveldown that works for watermelondb

Where communities thrive

People

Repo info

Activity