@abhaymaniyar unfortunately in Bleve it's pretty hard-coded that the segment files are on disk locally.
I have a newer library bluge (https://github.com/blugelabs/bluge) which is an experimental fork of bleve. It has support for a pluggable "directory" interface which removes this limitation. There has been interest expressed in the slack channel to add s3 support to bluge, but no work has started yet.
What I am a bit confused about right now is how "production ready" bluge is, or if should use bleve or bluge for a new project, generally speaking I am leaning towards bluge just because bleve seems a bit bloated with different indexes and query parsers and all, i would prefer the much slimmer bluge right now but not sure if thats a good idea
welcome @tulpenhaendler in my opinion, bluge is still only at developer preview release quality. it works for my use cases, but like bleve, it has unit tests, and some basic full-stack tests, but really lacks something more rigorous. in bleve, we have gotten by because Couchbase has invested in a considerable test suite for their product, which while not perfect, has functioned as a sort of proxy for that.
i will probably be making some announcements about bluge in the near future, but one thing is clear, it will need help from the community to become production ready, it is not something i will be able to do myself.
i appreciate you liking bluge being much leaner, that was one of my core goals when i started it
regarding the directory interface, it should be possible to do something with s3, but i suspect as currently implemented, you'll need some sort of local caching layer, and i'm not sure that bluge's use of segments will make that easy (not impossible either though)
for s3, i want to have lots of indexes shaded among multiple instances and they would just fetch the entire index once if the dont already have it local and upload snapshots periodically (so i dont actually need bluge to do anything s3 related), technically i guess s3 supports Range queries so you might even be able to implement something like ReadAt(offset,len) but imo it would be a strange use case where you need to use s3 like a filesystem....playing around on aws cost calculator - just the PUT requests for 5 writes/second are about the same price as a 500gb ebs drive per month...
I am talking about this repo: https://github.com/blugelabs/beer-search
here is a kitchen sink demo of GIOUI: https://gioui.org/files/wasm/kitchen/index.html
here si the kitchen sink code: https://github.com/gioui/gio-example/blob/main/kitchen/kitchen.go
Because beer search expects a FS, i will be using the golang FS wrapper that compiles to WASM and normal GO.
@gedw99 in general I try to keep the examples focused on the bleve/bluge aspects of the application. No matter which JS library we choose, or even choosing to go without one, it still gets in the way when we build web-based examples. Using a Go UI library may help some because the code is in Go, but it still hurts because it isn't how most apps would actually be built (today). So, for now I encourage you to build this if it is of interest to you, but I cannot say whether or not it would be accepted upstream. When you have something working that we can look at/use, please share it here again.
6 replies
Hi, I am wondering if there is an easy way to selectively index : for example to provide a database of words not to index at index creation time in order to speed it up and focus it on specific application needs. I can code it if need be, but would be interested in some pointers...
Hey Everyone, at the end of August I will be stepping back from the Bleve project. I am not sure if any of the core developers will monitor conversations here, consider the Gophers slack if you need assistance: https://groups.google.com/g/bleve/c/jghqnwh_VjQ/m/766ZGYqSAQAJ
Is this what's called a Facet?
For beer the facets are alcohol %
brewery name
so facets are things that are common to the data your indexing.
Does this help ?
Look here and it wil be obvious: https://github.com/blugelabs/beer-search
Facets are on the right of the image
In the GUi they are called Filters.
which is a good name for them
I see there's some interest in using S3 as a storage layer but I'm wondering if anyone has tried using EFS with Lambda? I'm trying to implement a "serverless" search service using Lambda and I think it might be possible with Bluge as it can have multiple processes accessing the same file. I'll give it a go with the beer-search demo that @gedw99 linked to and report back.
Hey @mousedownmike Interesting. Actually i am planning to do a similar thing with bluge, except store the data on the File system for Dekstop and Mobile, and the indexeddb for Browser. SO if you have the s3 prototype somewhere let me know... Will be possible from your work to also get it to work for the things i outlined
Hi all, I have a set of ~70 docs that I am indexing with Bleve (which is using the boltdb store by default). Indexing step works fine but when I do index.Open(indexPath), it takes forever and never returns anything.
Indexing and search operations are happening at two different times.
2 replies
@gedw99 If you are interested in storing indexes on object storage, please try this project.
https://github.com/mosuka/phalanx
https://github.com/mosuka/phalanx
