(If it's helpful, bleve dump gives
`Backindex DocId: `note1.org` Terms Entries: [field:0 ], Stored Entries: []
Key: 62 6e 6f 74 65 31 2e 6f 72 67
Value: 0a 02 08 00
Backindex DocId: `note2.org` Terms Entries: [field:0 ], Stored Entries: []
Key: 62 6e 6f 74 65 32 2e 6f 72 67
Value: 0a 02 08 00
Field: 0 Name: _all
Key: 66 00 00
Value: 5f 61 6c 6c ff
InternalStore - Key: _mapping (5f 6d 61 70 70 69 6e 67) Val: {"types":{"note":{"enabled":true,"dynamic":true,"properties":{"blocks":{"enabled":true,"dynamic":true,"properties":{"contents":{"enabled":true,"dynamic":true,"fields":[{"type":"text","analyzer":"en","store":true,"index":true,"include_term_vectors":true,"include_in_all":true,"docvalues":true}]}}},"title":{"enabled":true,"dynamic":true,"fields":[{"type":"text","analyzer":"en","store":true,"index":true,"include_term_vectors":true,"include_in_all":true,"docvalues":true}]}}}},"default_mapping":{"enabled":true,"dynamic":true},"type_field":"_type","default_type":"_default","default_analyzer":"standard","default_datetime_parser":"dateTimeOptional","default_field":"_all","store_dynamic":true,"index_dynamic":true,"docvalues_dynamic":true,"analysis":{}}which is what led me to suspect the mappings, I expected to see at least another field for title)
package main
import (
"fmt"
"github.com/blevesearch/bleve"
"log"
"os"
)
type Note struct {
Id string
Title string
}
func (n Note) Type() string {
return "note"
}
func main() {
os.RemoveAll("/tmp/idx")
englishMapping := bleve.NewTextFieldMapping()
englishMapping.Analyzer = "en"
blockMapping := bleve.NewDocumentMapping()
blockMapping.AddFieldMappingsAt("contents", englishMapping)
noteMapping := bleve.NewDocumentMapping()
noteMapping.AddFieldMappingsAt("title", englishMapping)
noteMapping.AddSubDocumentMapping("blocks", blockMapping)
indexMapping := bleve.NewIndexMapping()
indexMapping.AddDocumentMapping(Note{}.Type(), noteMapping)
idx, err := bleve.NewUsing("/tmp/idx", indexMapping, "scorch", "scorch", nil)
if err != nil {
log.Fatal(err)
}
err = idx.Index("note1.org", Note{Id: "note1.org", Title: "note1.org"})
if err != nil {
log.Fatal(err)
}
query := bleve.NewMatchQuery("note1.org")
search := bleve.NewSearchRequest(query)
searchResults, err := idx.Search(search)
if err != nil {
log.Fatal(err)
}
fmt.Println(searchResults)}
The upstream code has moved on a bit since then. But I am more than happy to fix or redo the PR if you would like any changes.
Thanks @mschoch , that actually is helpful - in that it gives me a starting point, and a birds I view of what the Field interface does.
I'll have a read through the existing Field implementations and see if I can figure it out from here.
It would be great it there were more extensive comments in document/field.go giving an overview and explaining the role of each of the methods.
An unrelated question - what is the status of the Scorch backend now? Is it ready for production use?
And how do we turn it on? Do we just set Config.DefaultIndexType = scorch.Name and specifybleve.NewUsing(path, mapping, "scorch", "scorch", nil)? Or are there other things we need to do?
How does performance compare with Rocksdb?
Thanks again for all the pointers. They will help me make sense of what the code is doing.
I think I will start with storing the IP addresses as the 16 byte ipv6 address and do an inefficient
full table scan for cidr matches.
On the other hand, cidr matches all have the same prefix, so if I can search for the lowest member and scan forward from there, I should be able to get efficient CIDR matches.
What happens if we want to be able to store multiple IPs in a field?
Or for that matter, multiple keywords in a text field?
Is this something that bleve handles naturally? Or should I generate multiple documents - if I have 5 IPs should I generate 5 copies of my document, one for each IP?
1 reply
