simerplaha on develop
- bump Scala & sbt versions and… (compare)
@algobardo hey mate, just pushed another release with Monix support. Added 2 examples here demoing how to use Task.
I've never worked with monix before so any suggestions on improving the examples or testing would help a lot.
Let me know how you go.
Hi @oridag, thank you. Supporting scala-native is something I'm looking forward to as well (#1).
I wish I could say that we are production ready now but we need to write more integration test-cases (#178) and finish few tasks relating to measuring and monitoring performance (specially #276).
Best case scenario is that we reach production readiness before new year otherwise we will definitely be production ready by early-mid next year.
Bag
implementation into your code. Proper release will happen when I win the battle against JVM's garbage collector. import cats.effect.IO
import cats.effect.unsafe.IORuntime
import swaydb.Bag.Async
import swaydb.serializers.Default._
import swaydb.{IO => SwayIO, _}
import scala.concurrent.{ExecutionContext, Future, Promise}
import scala.util.Failure
object Cats3Example extends App {
/**
* Cats-effect 3 async bag implementation
*/
implicit def apply(implicit runtime: IORuntime): swaydb.Bag.Async[IO] =
new Async[IO] { self =>
override def executionContext: ExecutionContext =
runtime.compute
override val unit: IO[Unit] =
IO.unit
override def none[A]: IO[Option[A]] =
IO.pure(Option.empty)
override def apply[A](a: => A): IO[A] =
IO(a)
override def map[A, B](a: IO[A])(f: A => B): IO[B] =
a.map(f)
override def transform[A, B](a: IO[A])(f: A => B): IO[B] =
a.map(f)
override def flatMap[A, B](fa: IO[A])(f: A => IO[B]): IO[B] =
fa.flatMap(f)
override def success[A](value: A): IO[A] =
IO.pure(value)
override def failure[A](exception: Throwable): IO[A] =
IO.fromTry(Failure(exception))
override def foreach[A](a: IO[A])(f: A => Unit): Unit =
f(a.unsafeRunSync())
def fromPromise[A](a: Promise[A]): IO[A] =
IO.fromFuture(IO(a.future))
override def complete[A](promise: Promise[A], a: IO[A]): Unit =
promise tryCompleteWith a.unsafeToFuture()
override def fromIO[E: SwayIO.ExceptionHandler, A](a: SwayIO[E, A]): IO[A] =
IO.fromTry(a.toTry)
override def fromFuture[A](a: Future[A]): IO[A] =
IO.fromFuture(IO(a))
override def suspend[B](f: => IO[B]): IO[B] =
IO.defer(f)
override def flatten[A](fa: IO[IO[A]]): IO[A] =
fa.flatMap(io => io)
}
implicit val runtime = IORuntime.global
val test =
for {
map <- memory.Map[Int, String, Nothing, IO]()
_ <- map.put(key = 1, value = "one")
value <- map.get(key = 1) //returns "one"
} yield {
println(s"value: $value")
}
test.unsafeRunSync()
}
PartitionId : TopicId : TopicOffset
I see you did a cats effect 3 release so I am guessing you won the battle with the garbage collector :-)
Oh the battle with the GC has been on for a while. SwayDB outperforms RocksDB when write workload is small-medium but on heavy compaction workloads longer GC pauses occur frequently so the battle is still on.
I would love to understand more about why you made swaydb.
There were many reasons to start but in general I felt that existing solutions always fell short one way or another. Thought a storage engine that (following general Scala philosophy) allowed simple data-structures to be composed easily to build more rich data-structures was needed. A company I used to work at got hefty monthly cloud bills for running ML training on large data so reducing these bills was also a major motivation.
My use case is http based messaging middleware with a large number of topics / mailboxes. where swaydb would be the underlying storage engine
I'm glad to hear that and would love to learn more. Is it a distributed system?
Wondering if there are any performance nobs I should turn.
There is heaps you can do here. The basic idea behind all configurations is that if something can determine Disk, CPU & RAM usage then it should be configurable. But I think this needs more documentation showing experiments with different settings and results.
the RocksDB code we create a RocksDB per day and at an atomic moment each day we drop the oldest rocks db from the end of partition list and add a new one to front of the list...
That's clever. Yep MultiMap
should make this very easy.
I'm sure you know this already but please note that SwayDB is not production ready yet. Quick status overview is that there a total of 2,897 test-cases (unit, stress, integration & performance tests) to ensure that nothing leads to incorrect data or data corruption but a solution for reducing GC pauses on heavy compaction workload is still pending.
So the general idea is the following
* mailboxes are lightweight and are created with a public and a private key
* you can write messages to a mailbox using the public key
* you can read messages from a mailbox using the private key
* best practice transport is to use the http methods they use server sent events and / or web sockets with a few other more esoteric mechanisms
* the idea is you can do all your RPC over this. So browsers can talk directly to browsers, servers to talk servers, a browser doesn't have to talk to only "it's" server but can be easily moved to talk to another server
* all mailboxes are just a sequence of messages ALWAYS stored to disk
* reading a mailbox is just saying what index to start at
* tailing a mailbox is an in memory transaction (lowers latency)
* work load is LOTS of mailboxes without any specific mailbox really getting a ton of IO though we have use cases where we push pulsar / kafka data through a mailbox and hence get high IO, those work fine you
* designed to be distributed but single server for now
we have tried a LOT of things to make this work over the years (various messaging servers most recently pulsar) and they all sort of fall over at some point... for example pulsar with a large number of topics / mailboxes all tailing thrashes really hard, like we can take down large pulsar clusters kind of thrashing.
In effect you can get around the infrastructure game (load balancing, etc, etc) but using this, which is what we do. We run some large customers using it and really avoid a TON of infrastrucutre headaches (think legacy systems, multiple clouds, things moving around all the time and multiple teams)... We run these systems with very small teams because of it. We deploy a service make sure it can reach the hermes server and everything else just works.
For debugging it is really cool. All I need are the two mailboxes of a conversation and I can re-construct the entire conversation... If you built your browser app properly can even do some ELM like things for replay...