Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Amrit Panesar
    @Neo-Desktop
    getting an err: panic: invalid page type: 0: 4
    this is triggered here:
            // open a new index
            mapping := bleve.NewIndexMapping()
            index, err := bleve.New("example.bleve", mapping)
            if err != nil {
                    fmt.Println(err)
                    return
            }
    was expecting this to "Just Work (tm)"
    Amrit Panesar
    @Neo-Desktop
    which actually bubbles up from:
    goroutine 1 [running]:
    github.com/etcd-io/bbolt.(*Cursor).search(0xc000145bb0, 0xc000145c10, 0x5, 0x20, 0x4)
            /home/neo/go/src/github.com/etcd-io/bbolt/cursor.go:250 +0x353
    github.com/etcd-io/bbolt.(*Cursor).seek(0xc000145bb0, 0xc000145c10, 0x5, 0x20, 0x0, 0x0, 0x0, 0x1, 0xc00029e000, 0x0, ...)
            /home/neo/go/src/github.com/etcd-io/bbolt/cursor.go:159 +0x7d
    github.com/etcd-io/bbolt.(*Bucket).Bucket(0xc000298018, 0xc000145c10, 0x5, 0x20, 0x5)
            /home/neo/go/src/github.com/etcd-io/bbolt/bucket.go:105 +0xd4
    github.com/etcd-io/bbolt.(*Tx).Bucket(...)
            /home/neo/go/src/github.com/etcd-io/bbolt/tx.go:101
    github.com/blevesearch/bleve/index/store/boltdb.(*Store).Reader(0xc000296000, 0xe2bd80, 0xc000081ef0, 0xa6a860, 0xc000296000)
            /home/neo/go/src/github.com/blevesearch/bleve/index/store/boltdb/store.go:123 +0xac
    github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).Open(0xc00013c380, 0x0, 0x0)
            /home/neo/go/src/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:315 +0x15f
    github.com/blevesearch/bleve.newIndexUsing(0x9ba0bf, 0xd, 0xa6eda0, 0xc00013c300, 0x9b970a, 0xb, 0x9b7c08, 0x6, 0x0, 0x0, ...)
            /home/neo/go/src/github.com/blevesearch/bleve/index_impl.go:109 +0x4ab
    github.com/blevesearch/bleve.New(...)
            /home/neo/go/src/github.com/blevesearch/bleve/index.go:257
    main.main()
            /home/neo/karaokefiles-server/main.go:12 +0x98
    exit status 2
    prog20901
    @prog20901
    Which is the best tool for generating javadoc in different formats ??? I want to generate javadoc with search engine.....Please advise...
    Frédéric G. MARAND
    @fgm
    Hello. I have a problem with some data: when the queried string appears several times in a document, I only get at most 1 fragment per field on matching documents. Others hits are not returned. My documents look like struct { Title string, Lines []string }, so I get at most 1 fragment on Title and 1 on Lines (assuming they match the query). How can I get all hits ?
    Kevin Langhans
    @klanghans
    Hello everybody
    Does anyone know the max filesize for a bleve index stored in boltdb ?
    Marlon Hendred
    @msquaredh
    so is bleve the lucene equivelent written in go? is there an bleve backed elasticsearch equvilent written in go?
    Kevin Langhans
    @klanghans
    Question #1 kind of
    Kevin Langhans
    @klanghans
    Question #2 you have to be more specific “bleve backed elastic search equivalent...” what do you mean by equivalent?
    Marlon Hendred
    @msquaredh
    for question #2 i mean like a standalone search engine that you could index data into and search/filter via REST apis that hadles index replication/sharding etc for you
    Kevin Langhans
    @klanghans
    As far as i have worked with bleve that is possible.
    I specifically chose bleve over elasticsearch because of the way lower memory requirements it has.
    I simply wrote asimple REST API in front of bleve and thats it.
    In regards of the sharding, i am nowhere near any requirements to shard right now, so i cant talk about that.
    Marlon Hendred
    @msquaredh
    @klanghans how big is the data set (doc count, disk space)? what is your read to write ratio to your bleve search app. what is the usecase?
    Kevin Langhans
    @klanghans
    @msquaredh documents: 5000, 90/10 read/write, disc space: 24MB, this is the setup for one customer, multiply that by 1000.
    Each customer has a confined userspace set with a separate bleve index on its own.
    @msquaredh we found that with such a small footprint, bleve is ideal to isolate userspaces without having to use containers, which removes all the container management hassle and overhead from the project
    the use case: learning video producers use our product to manage their customers to upload/edit videos to a video platform
    @msquaredh one thing we found is that bleve really needs some good i/o performance otherwise it stresses the device bus by a lot
    Håkon K. Eide
    @hkeide
    Hi, how do I pass a config map into an analyzer constructor? I can't find any examples doing this in the documentation or in the code base itself
    Håkon K. Eide
    @hkeide
    I think I figured it out: indexMapping.AddCustomAnalyzer() takes a config parameter
    Peter Mrekaj
    @mrekucci_gitlab
    Hi all, I'd like to ask whether the current version of bleve has an autocomplete feature? I saw some open issues regarding this: blevesearch/bleve#858 blevesearch/bleve#377 and some discussion where was suggested that the core have to be changed to address this functionality: https://groups.google.com/forum/#!topic/bleve/zd6vWSuCIEA
    If it isn't possible, what's the current state of affairs, is this on roadmap?
    Jörn Friedrich Dreyer
    @butonic
    I would like to index accounts and groups. The struct is generated using protobuf, but I can add the func BleveType() string to them. How can I add a type to the indexed document, so that I can build queries to only list accounts or groups? Or should I be using two indexes for that? Ideally I can later query for accounts and groups and get a mixed set of results, but for that I need to have a property in the index ... but how if I cannot add it to the protobuf?
    Hm, I could wrap the struct in my own struct ... with an additional type property ...
    xlzheng021
    @xlzheng021
    Hi All, could anyone help me with the Html CharFilter? I was trying to write my own Analyzer + html CharFilter, but it seems the html tags did not being stripe out:
    https://repl.it/@larrypage/FarawayKindheartedStartup#main.go
    I created my own Analyzer just directly copy the "en" analyzer, and also on top of that I added the html CharFilter
    Marty Schoch
    @mschoch
    @mandeepkhatry today you have to choose, whether you want to index age as text or number field (in the index), and that choice should be made based on the types of queries you want to support. if you need to do numeric range searches, you should index the field as a number. if you only need exact matches, you should index the field as text
    now, that just relates to how you set up your mapping, after making that choice, it will limit which types of input get handled
    if you choose text, only the string version would get indexed, if you choose number, only the numeric value would get indexed
    we're actually in the middle of discussing some aproaches to allow indexing numeric values as text, but today it is not possible, you would have to "fixup" your data manually before passing it to bleve
    @mandeepkhatry second question, no scorch is not the default, but it is what we recommend
    not sure about that last question, yes if you indexed age as a numeric field, you would do a numeric range query with min and max set to 10
    and call SetField on the query to "age"
    for these types of questions, it's often best to share a 20 line program that we can discuss the behavior of
    Mandeep Khatry
    @mandeepkhatry

    @mschoch I have a code https://play.golang.org/p/Iq6YfMF7LGk
    When I run my code, I get an error

    start batch indexing....
    panic: runtime error: index out of range [-1]
    
    goroutine 25 [running]:
    github.com/blevesearch/zap/v11.mergeStoredAndRemap.func2(0xc001483130, 0x9, 0x74, 0xc01181aeab, 0xc, 0xc, 0x0, 0x0, 0x0, 0x1)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:687 +0x579
    github.com/blevesearch/zap/v11.(*SegmentBase).visitDocument(0xc0083ca000, 0xc00041baa0, 0x1287, 0xc00038f618, 0x19, 0x0)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/segment.go:384 +0x383
    github.com/blevesearch/zap/v11.mergeStoredAndRemap(0xc007a80c80, 0xa, 0xa, 0xc007a80c30, 0xa, 0xa, 0xc004103380, 0xc016700000, 0x271a2, 0x271a2, ...)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:685 +0x47f
    github.com/blevesearch/zap/v11.MergeToWriter(0xc007a80c80, 0xa, 0xa, 0xc007a80c30, 0xa, 0xa, 0x400, 0xc004103350, 0xc00c0cd5c0, 0x203001, ...)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:132 +0x1ef
    github.com/blevesearch/zap/v11.mergeSegmentBases(0xc007a80c80, 0xa, 0xa, 0xc007a80c30, 0xa, 0xa, 0xc013fcfe90, 0x22, 0xc000000400, 0xc00c0cd5c0, ...)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:79 +0x24c
    github.com/blevesearch/zap/v11.(*ZapPlugin).Merge(0xfd0548, 0xc013f9ae60, 0xa, 0xa, 0xc007a80c30, 0xa, 0xa, 0xc013fcfe90, 0x22, 0xc00c0cd5c0, ...)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/zap/v11@v11.0.9/merge.go:54 +0x14f
    github.com/blevesearch/bleve/index/scorch.(*Scorch).planMergeAtSnapshot(0xc0002c2000, 0xb68640, 0xc0000c2040, 0xc00030a000, 0xc00e8d4e00, 0x0, 0x0)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/bleve@v1.0.9/index/scorch/merge.go:321 +0xad9
    github.com/blevesearch/bleve/index/scorch.(*Scorch).mergerLoop(0xc0002c2000)
            /home/mandeep/work/pkg/mod/github.com/blevesearch/bleve@v1.0.9/index/scorch/merge.go:68 +0x2c7
    created by github.com/blevesearch/bleve/index/scorch.(*Scorch).Open
            /home/mandeep/work/pkg/mod/github.com/blevesearch/bleve@v1.0.9/index/scorch/scorch.go:206 +0x125
    exit status 2

    Any problem on this simple code?

    Marty Schoch
    @mschoch
    @mandeepkhatry I don't see any obvious problem, reproduced the issue. I have opened an issue to track it here: blevesearch/zap#28
    Marty Schoch
    @mschoch
    @mandeepkhatry Ah I see the issue now. Bleve only supports 2^16 fields, however this code exceeds that (using i in the field name).
    Mandeep Khatry
    @mandeepkhatry
    @mschoch Thank you for the reply.
    Mandeep Khatry
    @mandeepkhatry
    @mschoch I don't think its the problem with field. After running the iteration, the error happened at i =350349 with fields "name350349" and "age350349". When I index a document with the fields (only one document), it works
    I tried with document {"name350349" : "TestName350439", "age350349" : 350449"} only single document (without loop testing this particular document only), it works fine.
    Mandeep Khatry
    @mandeepkhatry
    Sorry I may have misinterpreted. Do u mean its 2^16 different fields (i.e indexing 65536 different fields ) in single store? If so, What if I want more than different 2^16 fields?
    Mandeep Khatry
    @mandeepkhatry
    When I run the same program with data["age"+strconv.Itoa(i)] = i only
    (comment the line "data["name"+strconv.Itoa(i)] = "TestName" + strconv.Itoa(i)")
    it works
    Marty Schoch
    @mschoch
    the index organizes information by field, there is a limit of 2^16 fields in the index
    there is currently no way around this, the file format uses a fixed-width 16-bit integer for field identifier, and all the surrounding code makes the same assumption
    because of the way you're using fields unique to documents, the problem only shows up when we merge 2 segments, and the resulting segment would require > 2^16 fields in the same segment
    i suspect that is why it may work in the most recent example you gave, even though you used more fields than allowed, it doesn't actually crash until a merge happens
    Marty Schoch
    @mschoch
    You'd have to share the whole mapping. Unfortunately, the most common issue is that basic problems in the mapping mean it isn't even getting used.
    Mandeep Khatry
    @mandeepkhatry

    @mschoch Thanks for being so active in the community.
    I have a data with field "age" : "22" 22 as string with ID 1 and another data with field "age" : 22 as number with ID 2.
    My searching
    bleve.NewQueryStringQuery(age:22)
    I get ids both [1,2]
    How can I use query to query only number age and only string age?

    Also seperate between string "false" and boolean false in QueryStringQuery

    Marty Schoch
    @mschoch
    @mandeepkhatry so you probably need to build your own query string capability, we basically engineered the query string to hide those differences, not expose them
    Mandeep Khatry
    @mandeepkhatry

    @mschoch
    For matching queries related to numeric to match data with integer age 22
    bleve.NewQueryStringQuery("age:>=22 age:<=22") seems to work
    FOr string to match data with string age "22"
    bleve.NewQueryStringQuery("age:'22' ")
    What do u think on this?

    Or is there anything something like for boolean we use isEmployee:T* . Is there any anything for numeric without doing bleve.NewQueryStringQuery("age:>=22 age:<=22") ?