so, I will say a little about existing solutions for MongoDB in Scala, and why I started to develop MongoQuery, before cover the implementation details.
the problem is that I didn't find the way to construct BsonDocument in macro from the List[(String, Any)].
val elements: List[(String, Any)] = List("name" -> "name", "age" -> 3, "salary" -> 4.0)
val document: BsonDocument = BsonDocument.from(elements)
you can construct a BsonDocument from (String, Any) tuples like above.
if you allow, i can paste the link of my query here ?
Or you can ask in the reactivemongo google group
This room is just for tepkin
that's why, i taken the permission from you
If anyone is interested I have opened a room for my new project: elastic-streams https://gitter.im/fehmicansaglam/elastic-streams
I am trying to come up with an implementation of reactive streams for elasticsearch
hey @fehmicansaglam . Sorry that it took so long to report back. The collection sink api looks just as expected. very nice, thank you. I had no luck yet running this in a combined example (like having a flow stdin->collection.sink ; collection.find(tailable=true) -> stdout) but I'll give you a link once its running. might be a nice example.
I am still wondering if one could make the collection.find() returning directly a
Source instead of returning a Future[Source]. In the case when the future would fail, you could just let the stream itself fail directly. This could enable the use of stream supervision for recoverable failures (like when the mongo is not available during flow materialization, recover with a restart). In the end, Sources are easier to compose.
I agree that it is easier to use the api right now
Mhm weird. Sink seems to work fine but the MongoCursor seem to receive a reply with a cursorId 0 (as of documentation that states that indicates a dead cursor, whatever that means) and thus eagerly closes the stream. Would you be fine if I'd add ResponseFlags to the reply class?
What do you mean by ResponseFlags? Could you point me to the documentation?
The response flags are documented in http://docs.mongodb.org/meta-driver/latest/legacy/mongodb-wire-protocol/#op-reply . They already a field
flags in https://github.com/fehmicansaglam/tepkin/blob/master/tepkin/src/main/scala/net/fehmicansaglam/tepkin/protocol/message/Reply.scala but are not parsed (but maybe I am in the wrong version of the wire protocol).
I think I have misunderstood the tailable cursors then. But why would the mongo give me a dead cursor for a tailable cursor query on an empty collection if I want to query all elements that will arrive there in the near future. I thought that is what tailable cursors / capped collections are made for.
Maybe there is a way to handle empty collections
The dead cursor on empty collection seem to be an off by one corner case of the dead cursor "the cyclic log buffer has overwritten your current read position" - like in "the producer is faster than the consumer". The tailable cursor documentation state that the user should explicitply deal with such errors (in their example, using a for loop). We maybe could fail the source if we receive a dead cursor on tailable
MongoCursors (it should never complete normally). if (cursorID == 0 && tailable) onError(new DeadCursorException()). The user could then decide to use a proper failure policy (restart or fail the whole stream, or switch to another source or whatever)
I think we can rely on stream supervision such that the user can decide if she wants to restart/stop/fail downstream (see http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M5/scala/stream-error.html)
The restart should just restart the stage (create an new
MongoCursor actor)
BTW I have added . notation support for keys as can be seen here: https://github.com/fehmicansaglam/tepkin/blob/master/bson%2Fsrc%2Ftest%2Fscala%2Fnet%2Ffehmicansaglam%2Fbson%2FBsonSpec.scala#L40