Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
Repo info
    Marlon Hendred
    @klanghans how big is the data set (doc count, disk space)? what is your read to write ratio to your bleve search app. what is the usecase?
    Kevin Langhans
    @msquaredh documents: 5000, 90/10 read/write, disc space: 24MB, this is the setup for one customer, multiply that by 1000.
    Each customer has a confined userspace set with a separate bleve index on its own.
    @msquaredh we found that with such a small footprint, bleve is ideal to isolate userspaces without having to use containers, which removes all the container management hassle and overhead from the project
    the use case: learning video producers use our product to manage their customers to upload/edit videos to a video platform
    @msquaredh one thing we found is that bleve really needs some good i/o performance otherwise it stresses the device bus by a lot
    Håkon K. Eide
    Hi, how do I pass a config map into an analyzer constructor? I can't find any examples doing this in the documentation or in the code base itself
    Håkon K. Eide
    I think I figured it out: indexMapping.AddCustomAnalyzer() takes a config parameter
    Peter Mrekaj
    Hi all, I'd like to ask whether the current version of bleve has an autocomplete feature? I saw some open issues regarding this: blevesearch/bleve#858 blevesearch/bleve#377 and some discussion where was suggested that the core have to be changed to address this functionality: https://groups.google.com/forum/#!topic/bleve/zd6vWSuCIEA
    If it isn't possible, what's the current state of affairs, is this on roadmap?
    Jörn Friedrich Dreyer
    I would like to index accounts and groups. The struct is generated using protobuf, but I can add the func BleveType() string to them. How can I add a type to the indexed document, so that I can build queries to only list accounts or groups? Or should I be using two indexes for that? Ideally I can later query for accounts and groups and get a mixed set of results, but for that I need to have a property in the index ... but how if I cannot add it to the protobuf?
    Hm, I could wrap the struct in my own struct ... with an additional type property ...
    Hi All, could anyone help me with the Html CharFilter? I was trying to write my own Analyzer + html CharFilter, but it seems the html tags did not being stripe out:
    I created my own Analyzer just directly copy the "en" analyzer, and also on top of that I added the html CharFilter
    Marty Schoch
    @mandeepkhatry today you have to choose, whether you want to index age as text or number field (in the index), and that choice should be made based on the types of queries you want to support. if you need to do numeric range searches, you should index the field as a number. if you only need exact matches, you should index the field as text
    now, that just relates to how you set up your mapping, after making that choice, it will limit which types of input get handled
    if you choose text, only the string version would get indexed, if you choose number, only the numeric value would get indexed
    we're actually in the middle of discussing some aproaches to allow indexing numeric values as text, but today it is not possible, you would have to "fixup" your data manually before passing it to bleve
    @mandeepkhatry second question, no scorch is not the default, but it is what we recommend
    not sure about that last question, yes if you indexed age as a numeric field, you would do a numeric range query with min and max set to 10
    and call SetField on the query to "age"
    for these types of questions, it's often best to share a 20 line program that we can discuss the behavior of
    Mandeep Khatry

    @mschoch I have a code https://play.golang.org/p/Iq6YfMF7LGk
    When I run my code, I get an error

    start batch indexing....
    panic: runtime error: index out of range [-1]
    goroutine 25 [running]:
    github.com/blevesearch/zap/v11.mergeStoredAndRemap.func2(0xc001483130, 0x9, 0x74, 0xc01181aeab, 0xc, 0xc, 0x0, 0x0, 0x0, 0x1)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:687 +0x579
    github.com/blevesearch/zap/v11.(*SegmentBase).visitDocument(0xc0083ca000, 0xc00041baa0, 0x1287, 0xc00038f618, 0x19, 0x0)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/segment.go:384 +0x383
    github.com/blevesearch/zap/v11.mergeStoredAndRemap(0xc007a80c80, 0xa, 0xa, 0xc007a80c30, 0xa, 0xa, 0xc004103380, 0xc016700000, 0x271a2, 0x271a2, ...)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:685 +0x47f
    github.com/blevesearch/zap/v11.MergeToWriter(0xc007a80c80, 0xa, 0xa, 0xc007a80c30, 0xa, 0xa, 0x400, 0xc004103350, 0xc00c0cd5c0, 0x203001, ...)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:132 +0x1ef
    github.com/blevesearch/zap/v11.mergeSegmentBases(0xc007a80c80, 0xa, 0xa, 0xc007a80c30, 0xa, 0xa, 0xc013fcfe90, 0x22, 0xc000000400, 0xc00c0cd5c0, ...)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:79 +0x24c
    github.com/blevesearch/zap/v11.(*ZapPlugin).Merge(0xfd0548, 0xc013f9ae60, 0xa, 0xa, 0xc007a80c30, 0xa, 0xa, 0xc013fcfe90, 0x22, 0xc00c0cd5c0, ...)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:54 +0x14f
    github.com/blevesearch/bleve/index/scorch.(*Scorch).planMergeAtSnapshot(0xc0002c2000, 0xb68640, 0xc0000c2040, 0xc00030a000, 0xc00e8d4e00, 0x0, 0x0)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/bleve@v1.0.9/index/scorch/merge.go:321 +0xad9
            /home/mandeep/work/pkg/mod/github.com/blevesearch/bleve@v1.0.9/index/scorch/merge.go:68 +0x2c7
    created by github.com/blevesearch/bleve/index/scorch.(*Scorch).Open
            /home/mandeep/work/pkg/mod/github.com/blevesearch/bleve@v1.0.9/index/scorch/scorch.go:206 +0x125
    exit status 2

    Any problem on this simple code?

    Marty Schoch
    @mandeepkhatry I don't see any obvious problem, reproduced the issue. I have opened an issue to track it here: blevesearch/zap#28
    Marty Schoch
    @mandeepkhatry Ah I see the issue now. Bleve only supports 2^16 fields, however this code exceeds that (using i in the field name).
    Mandeep Khatry
    @mschoch Thank you for the reply.
    Mandeep Khatry
    @mschoch I don't think its the problem with field. After running the iteration, the error happened at i =350349 with fields "name350349" and "age350349". When I index a document with the fields (only one document), it works
    I tried with document {"name350349" : "TestName350439", "age350349" : 350449"} only single document (without loop testing this particular document only), it works fine.
    Mandeep Khatry
    Sorry I may have misinterpreted. Do u mean its 2^16 different fields (i.e indexing 65536 different fields ) in single store? If so, What if I want more than different 2^16 fields?
    Mandeep Khatry
    When I run the same program with data["age"+strconv.Itoa(i)] = i only
    (comment the line "data["name"+strconv.Itoa(i)] = "TestName" + strconv.Itoa(i)")
    it works
    Marty Schoch
    the index organizes information by field, there is a limit of 2^16 fields in the index
    there is currently no way around this, the file format uses a fixed-width 16-bit integer for field identifier, and all the surrounding code makes the same assumption
    because of the way you're using fields unique to documents, the problem only shows up when we merge 2 segments, and the resulting segment would require > 2^16 fields in the same segment
    i suspect that is why it may work in the most recent example you gave, even though you used more fields than allowed, it doesn't actually crash until a merge happens
    Marty Schoch
    You'd have to share the whole mapping. Unfortunately, the most common issue is that basic problems in the mapping mean it isn't even getting used.
    Mandeep Khatry

    @mschoch Thanks for being so active in the community.
    I have a data with field "age" : "22" 22 as string with ID 1 and another data with field "age" : 22 as number with ID 2.
    My searching
    I get ids both [1,2]
    How can I use query to query only number age and only string age?

    Also seperate between string "false" and boolean false in QueryStringQuery

    Marty Schoch
    @mandeepkhatry so you probably need to build your own query string capability, we basically engineered the query string to hide those differences, not expose them
    Mandeep Khatry

    For matching queries related to numeric to match data with integer age 22
    bleve.NewQueryStringQuery("age:>=22 age:<=22") seems to work
    FOr string to match data with string age "22"
    bleve.NewQueryStringQuery("age:'22' ")
    What do u think on this?

    Or is there anything something like for boolean we use isEmployee:T* . Is there any anything for numeric without doing bleve.NewQueryStringQuery("age:>=22 age:<=22") ?

    Hi, Do any one knows how can I update part of a document?
    Marty Schoch
    @current1990 the design of bleve does not allow for partial updates, internally all fields from a document are linked to a document number, and when an update takes places, the previous document number is obsoleted
    Rupesh Kumar Sahu
    How can I perform date range searches by using the >, >=, <, and <= operators ?
    @mschoch Can i use NewMatchQuery for search date and time ?
    Marty Schoch
    i replied to your message on google group
    there was no need to send it 3 times, first time senders required approval
    John Moss

    Hello, I expect I'm making some beginner mistakes here with the mappers... can someone take a look at this?

    type Note struct {
      id string
      title string
      blocks []BasicBlock
    func (n Note) Type() string {
      return "note"
    type BasicBlock struct {
      id string
      contents string
    func (b BasicBlock) Type() string {
      return "block"
    englishMapping := bleve.NewTextFieldMapping()
    englishMapping.Analyzer = "en"
    blockMapping := bleve.NewDocumentMapping()
    blockMapping.AddFieldMappingsAt("contents", englishMapping)
    noteMapping := bleve.NewDocumentMapping()
    noteMapping.AddFieldMappingsAt("title", englishMapping)
    noteMapping.AddSubDocumentMapping("blocks", blockMapping)
    indexMapping := bleve.NewIndexMapping()
    indexMapping.AddDocumentMapping(Note{}.Type(), noteMapping)

    All queries return zero results except MatchAllQueries. Ideas?

    John Moss
    (I'm index.Index()ing Note structs)
    Marty Schoch
    @jmoss20 can you confirm you're indexing Note structs and not *Note pointers?
    John Moss
    an even more minimal example: index.Index("note1.org", Note{id: "note1.org", title: "note1.org"})
    query := bleve.NewMatchQuery("note1.org")
    search := bleve.NewSearchRequest(query)
    searchResults, err := index.Search(search)
    -> No matches