Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Mandeep Khatry
    @mandeepkhatry
    Sorry I may have misinterpreted. Do u mean its 2^16 different fields (i.e indexing 65536 different fields ) in single store? If so, What if I want more than different 2^16 fields?
    Mandeep Khatry
    @mandeepkhatry
    When I run the same program with data["age"+strconv.Itoa(i)] = i only
    (comment the line "data["name"+strconv.Itoa(i)] = "TestName" + strconv.Itoa(i)")
    it works
    Marty Schoch
    @mschoch
    the index organizes information by field, there is a limit of 2^16 fields in the index
    there is currently no way around this, the file format uses a fixed-width 16-bit integer for field identifier, and all the surrounding code makes the same assumption
    because of the way you're using fields unique to documents, the problem only shows up when we merge 2 segments, and the resulting segment would require > 2^16 fields in the same segment
    i suspect that is why it may work in the most recent example you gave, even though you used more fields than allowed, it doesn't actually crash until a merge happens
    Marty Schoch
    @mschoch
    You'd have to share the whole mapping. Unfortunately, the most common issue is that basic problems in the mapping mean it isn't even getting used.
    Mandeep Khatry
    @mandeepkhatry

    @mschoch Thanks for being so active in the community.
    I have a data with field "age" : "22" 22 as string with ID 1 and another data with field "age" : 22 as number with ID 2.
    My searching
    bleve.NewQueryStringQuery(age:22)
    I get ids both [1,2]
    How can I use query to query only number age and only string age?

    Also seperate between string "false" and boolean false in QueryStringQuery

    Marty Schoch
    @mschoch
    @mandeepkhatry so you probably need to build your own query string capability, we basically engineered the query string to hide those differences, not expose them
    Mandeep Khatry
    @mandeepkhatry

    @mschoch
    For matching queries related to numeric to match data with integer age 22
    bleve.NewQueryStringQuery("age:>=22 age:<=22") seems to work
    FOr string to match data with string age "22"
    bleve.NewQueryStringQuery("age:'22' ")
    What do u think on this?

    Or is there anything something like for boolean we use isEmployee:T* . Is there any anything for numeric without doing bleve.NewQueryStringQuery("age:>=22 age:<=22") ?

    current1990
    @current1990
    Hi, Do any one knows how can I update part of a document?
    Marty Schoch
    @mschoch
    @current1990 the design of bleve does not allow for partial updates, internally all fields from a document are linked to a document number, and when an update takes places, the previous document number is obsoleted
    current1990
    @current1990
    thank
    Rupesh Kumar Sahu
    @rupeshkumarsahu
    How can I perform date range searches by using the >, >=, <, and <= operators ?
    @mschoch Can i use NewMatchQuery for search date and time ?
    Marty Schoch
    @mschoch
    i replied to your message on google group
    there was no need to send it 3 times, first time senders required approval
    John Moss
    @jmoss20

    Hello, I expect I'm making some beginner mistakes here with the mappers... can someone take a look at this?

    type Note struct {
      id string
      title string
      blocks []BasicBlock
    }
    
    func (n Note) Type() string {
      return "note"
    }
    
    type BasicBlock struct {
      id string
      contents string
    }
    
    func (b BasicBlock) Type() string {
      return "block"
    }
    
    englishMapping := bleve.NewTextFieldMapping()
    englishMapping.Analyzer = "en"
    
    blockMapping := bleve.NewDocumentMapping()
    blockMapping.AddFieldMappingsAt("contents", englishMapping)
    
    noteMapping := bleve.NewDocumentMapping()
    noteMapping.AddFieldMappingsAt("title", englishMapping)
    noteMapping.AddSubDocumentMapping("blocks", blockMapping)
    
    indexMapping := bleve.NewIndexMapping()
    indexMapping.AddDocumentMapping(Note{}.Type(), noteMapping)

    All queries return zero results except MatchAllQueries. Ideas?

    John Moss
    @jmoss20
    (I'm index.Index()ing Note structs)
    Marty Schoch
    @mschoch
    @jmoss20 can you confirm you're indexing Note structs and not *Note pointers?
    John Moss
    @jmoss20
    yep
    an even more minimal example: index.Index("note1.org", Note{id: "note1.org", title: "note1.org"})
    query := bleve.NewMatchQuery("note1.org")
    search := bleve.NewSearchRequest(query)
    searchResults, err := index.Search(search)
    -> No matches
    John Moss
    @jmoss20

    (If it's helpful, bleve dump gives

    `Backindex DocId: `note1.org` Terms Entries: [field:0 ], Stored Entries: []
    Key:   62 6e 6f 74 65 31 2e 6f 72 67
    Value: 0a 02 08 00
    
    Backindex DocId: `note2.org` Terms Entries: [field:0 ], Stored Entries: []
    Key:   62 6e 6f 74 65 32 2e 6f 72 67
    Value: 0a 02 08 00
    
    Field: 0 Name: _all
    Key:   66 00 00
    Value: 5f 61 6c 6c ff
    
    InternalStore - Key: _mapping (5f 6d 61 70 70 69 6e 67) Val: {"types":{"note":{"enabled":true,"dynamic":true,"properties":{"blocks":{"enabled":true,"dynamic":true,"properties":{"contents":{"enabled":true,"dynamic":true,"fields":[{"type":"text","analyzer":"en","store":true,"index":true,"include_term_vectors":true,"include_in_all":true,"docvalues":true}]}}},"title":{"enabled":true,"dynamic":true,"fields":[{"type":"text","analyzer":"en","store":true,"index":true,"include_term_vectors":true,"include_in_all":true,"docvalues":true}]}}}},"default_mapping":{"enabled":true,"dynamic":true},"type_field":"_type","default_type":"_default","default_analyzer":"standard","default_datetime_parser":"dateTimeOptional","default_field":"_all","store_dynamic":true,"index_dynamic":true,"docvalues_dynamic":true,"analysis":{}}

    which is what led me to suspect the mappings, I expected to see at least another field for title)

    Marty Schoch
    @mschoch
    ah i see, sorry i didn't notice it sooner, the fields must be exported (as bleve uses reflection to examine your documents, it cannot see unexported fields)
    you can name the fields in your index lower-case if you want
    Marty Schoch
    @mschoch
    @jmoss20 here it is, hacked up a bit (and using scorch, not the old default upsidedown)
    ```

    package main

    import (
    "fmt"
    "github.com/blevesearch/bleve"
    "log"
    "os"
    )

    type Note struct {
    Id string
    Title string
    }

    func (n Note) Type() string {
    return "note"
    }

    func main() {

    os.RemoveAll("/tmp/idx")
    
    englishMapping := bleve.NewTextFieldMapping()
    englishMapping.Analyzer = "en"
    
    blockMapping := bleve.NewDocumentMapping()
    blockMapping.AddFieldMappingsAt("contents", englishMapping)
    
    noteMapping := bleve.NewDocumentMapping()
    noteMapping.AddFieldMappingsAt("title", englishMapping)
    noteMapping.AddSubDocumentMapping("blocks", blockMapping)
    
    indexMapping := bleve.NewIndexMapping()
    indexMapping.AddDocumentMapping(Note{}.Type(), noteMapping)
    
    idx, err := bleve.NewUsing("/tmp/idx", indexMapping, "scorch", "scorch", nil)
    if err != nil {
        log.Fatal(err)
    }
    
    err = idx.Index("note1.org", Note{Id: "note1.org", Title: "note1.org"})
    if err != nil {
        log.Fatal(err)
    }
    
    query := bleve.NewMatchQuery("note1.org")
    search := bleve.NewSearchRequest(query)
    searchResults, err := idx.Search(search)
    if err != nil {
        log.Fatal(err)
    }
    
    fmt.Println(searchResults)

    }

    gitter so terrible
    John Moss
    @jmoss20
    ahhh, thanks! Haven't touched go before, didn't realize uppercase/lowercase was significant
    appreciate the help
    Marty Schoch
    @mschoch
    sure np
    Amnon
    @amnonbc
    Hi all, we added an IPRange mapper to bleve. Is this something which you would be prepared to consider upstreaming? Should I submit a PR?
    Amnon
    @amnonbc
    Looking though the history, I can see that one of my predecessors did submit a PR for this a while back blevesearch/bleve#644.
    The upstream code has moved on a bit since then. But I am more than happy to fix or redo the PR if you would like any changes.
    Amnon
    @amnonbc
    I see that it has also been discussed on the mailing list https://groups.google.com/g/bleve/c/FBiUWJKWMZg/m/aUrXqGfxCgAJ
    Marty Schoch
    @mschoch
    yeah, i think that commentary captures my thoughts
    i would expect a new field type to encode all necessary information into a single field
    and then some new query or set of queries that operate on that data
    Amnon
    @amnonbc
    Thanks @mschoch, that definitely makes sense. I'll pick up the PR and make these changes. I would also like to add IPrange queries into the query string language if I can find a syntax you are happy with.
    Amnon
    @amnonbc
    I had a look at adding an IpField type. But I am missing some context about what Fields are for, and what their methods do. The comments in field.go are a bit terse, and the unit tests in the document directory don't really demonstrate what functionality is required. I am having trouble getting what the Analyze method is doing. Do you have any higher level explanation what the field types do?
    Marty Schoch
    @mschoch
    field types are resonsible for creating some representation of the data inside the index
    the index is an inverted index of terms
    for textual fields the Analyze() method splits the "full text" into the pieces, terms
    numeric, date, geo, etc typically do some artificial manipulation of the data
    i realize this isn't very helpful, but honestly not sure where to begin
    Amnon
    @amnonbc

    Thanks @mschoch , that actually is helpful - in that it gives me a starting point, and a birds I view of what the Field interface does.
    I'll have a read through the existing Field implementations and see if I can figure it out from here.

    It would be great it there were more extensive comments in document/field.go giving an overview and explaining the role of each of the methods.

    An unrelated question - what is the status of the Scorch backend now? Is it ready for production use?
    And how do we turn it on? Do we just set Config.DefaultIndexType = scorch.Name and specify
    bleve.NewUsing(path, mapping, "scorch", "scorch", nil)? Or are there other things we need to do?
    How does performance compare with Rocksdb?

    Marty Schoch
    @mschoch
    the reason there wont be more docs added is that the interface no longer makes sense