Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jun 18 02:56

    simerplaha on develop

    - bump Scala & sbt versions and… (compare)

  • Jun 02 10:25
    simerplaha commented #360
  • Jun 02 10:24
    simerplaha edited #317
  • Jun 02 10:23
    simerplaha commented #360
  • Jun 01 09:42
    simerplaha labeled #360
  • Jun 01 09:41
    simerplaha commented #360
  • Jun 01 08:57
    simerplaha commented #360
  • May 31 15:58
    hicolour edited #360
  • May 31 15:49
    hicolour opened #360
  • Apr 16 10:59
    simerplaha edited #359
  • Apr 11 14:58
    simerplaha edited #359
  • Apr 11 11:58
    simerplaha edited #359
  • Apr 11 11:56
    simerplaha edited #359
  • Apr 11 11:55
    simerplaha edited #359
  • Apr 11 11:55
    simerplaha labeled #359
  • Apr 11 11:55
    simerplaha opened #359
  • Mar 29 14:21
    simerplaha labeled #358
  • Mar 29 14:20
    simerplaha commented #358
  • Mar 29 14:19
    simerplaha commented #358
  • Mar 29 13:25
    hicolour opened #358
Simer Plaha
@simerplaha
This message was deleted
Simer Plaha
@simerplaha

Hi Frank, I haven't looked at setting up travis yet.

Random inserts are slower than sequential because of insertion cost of inserting randomly into skipList. That's just how skipLists are. In Level0 after key-values get written to a .log file they get added to an in-memory skipList. Sequential insertion would just append to the end of the skipList and adding new upper Levels (in the skipList) as required, but random insertions result in modifying and re-linking (similar to linkedList) multiple keys in the skipList and sometimes multiple Levels (skipList hierarchies).

You can run a simple test with java.util.concurrent.ConcurrentSkipListMap to see the performance difference.

By data structure I assuming you mean the 'file formats'. I don't think the current file formats are documented anywhere. The overview of what Map files (.log) and Segment files (.seg) are can be found on the website. The actual file formats needs to be documented. Have a look at the documentation at http://www.swaydb.io/terminology/ which is an overview of each file and the role they play in Levels which should give you a good understanding of how the Levels are structured.

Thank you for the chart :+1:

Simer Plaha
@simerplaha

Also why is the difference getting less when moving from 2-level in-memory towards 8-level regular file?

As you know RAM is a lot faster than MMAP files and MMAP files are faster than regular files (java.io.FileChannel).

The benchmarks should also show performance results when the databases cache is partly populated. As you read key-values they get cached in-memory and performance increases drastically. Reads get a lot faster as the database is running because the most read keys are in-memory which other reads piggy back off to save disk seeks.

Simer Plaha
@simerplaha
Please let me know if things are unclear.
Frank Rosner
@FRosner

As you know RAM is a lot faster than MMAP files and MMAP files are faster than regular files (java.io.FileChannel).

This is clear to me, of course. I was just wondering why the relative performance penalty becomes less for regular files than for MMAP files / in-memory. Do you know what I mean?

I now also understand that you are using skip lists as data structures for the in-memory levels. What are you using for storing on disk? B trees? SSTables? Some other structure?

Hi Frank, I haven't looked at setting up travis yet.

Ok I will see if I can find the time. What should travis run? sbt test?

Thank you for the chart :thumbsup:

I am currently writing a blog post series about data structures to refresh my knowledge. Is it ok if I use your numbers to visualize the effects of compaction and different persistence levels on the throughput? That's where the graph was going to go. Here is the latest post of the series: Read Efficient Data Structures. I am currently working on Update Efficient Data Structures.

Simer Plaha
@simerplaha

What are you using for storing on disk? B trees? SSTables? Some other structure?

I think its similar to SSTables. Here is an overview of the format - the Segment files store bytes in 3 groups.
1- Values - Top of the files stores all values bytes.
2 - Index - Keys stored in sorted order with other metadata related to the key such as id, compression info, value offset, TTL etc.
3 - Footer - stores bloomfilter & other file format information like key-value count, CRC, index offsets.

Ok I will see if I can find the time. What should travis run? sbt test?

Yep sbt test is correct.

Is it ok if I use your numbers to visualize the effects of compaction and different persistence levels on the throughput?

Yes of course that's OK. Looking forward to your blog post :+1:

... Here is the latest post of the series: Read Efficient Data Structures. I am currently working on Update Efficient Data Structures.

I had a quick read and it looks very interesting. I will have a proper read soon.

Simer Plaha
@simerplaha

The reason I used a custom format to get better compression.
A test inserting 10000000 key-values with compression and MMAP disabled resulted in following disk usage.

  • LevelDB - 200.3 mb
  • RocksDB - 197.6 mb
  • SwayDB - 149.5 mb

The above is just a quick test. I might be missing something from my test to tune LevelDB & RocksDB. It will be interesting to see the disk usage after releasing version 0.4 which will have better compressed file format and also support for LZ4.

Frank Rosner
@FRosner
Great! Thanks for sharing this awesome project!
Simer Plaha
@simerplaha
No worries :+1:
Simer Plaha
@simerplaha
Thank you for your awesome feedback :- )
Simer Plaha
@simerplaha

@/all Hey guys just released version 0.3. This release contains support for expire API (TTL), update & improved batching for atomic writes. Here is a sample of some of the new API.

//put & expire a key-value after a day
db.put(key = 1, value = "one", expireAfter = 1.day)
//or expire a range of key-values after an hour
db.expire(from = 1, to = 1000, after = 1.hour)
//update values without altering the already set expiration
db.update(from = 1, to = 1000, value = "value updated")
//or update a single value
db.update(key = 1, value = "value updated")
//fetch the expiration deadline for a key
db.expiration(key = 1)
//fetch time left until the key's expiration
db.timeLeft(key = 1)

Here is a whole list of write AP and read API.

I'm going to start working on getting compression ready for v0.4. If you think of any features you think we should add please do bring it up.

Other than that, all the core APIs are done. Any new features can be built as extension libraries. Let me know what you think.
Frank Rosner
@FRosner
I don't really have any strong opinions but great work releasing 0.3!
Also I couldn't find the time to look into the travis build :(
Simer Plaha
@simerplaha

Thank Frank.
No worries whenever you have time is OK. We will eventually get to it :)

Loved your blog posts BTW :+1:

Simer Plaha
@simerplaha

@/all Just pushed a big commit to support compression. Both LZ4 and Snappy libraries are full supported. It's part of the grouping strategy you can read up here .

Segment file format documentation
Grouping format documentation

Will do a release after simerplaha/SwayDB#15

Frank Rosner
@FRosner
Yeah :thumbsup:
Simer Plaha
@simerplaha
@/all Just released v0.4. I've added a topic to scala-lang with a brief outlining the release.
Peter C.
@touhonoob
Hello There, Do you guys have any stats on the storage footprint of SwayDB in comparison to LevelDB or LMDB?
in terms of mmap file
Simer Plaha
@simerplaha

Hi @touhonoob, just ran some quick space usages tests on 10 million key-values with compression disabled for both LevelDB and SwayDB.

SwayDB provides two storage formats Map and Set. So I've added space usage for both.

The following shows the total size of sstables created by LevelDB and segments files by SwayDB.

Space usage when keys and values are unique integers

Key -> Value = 1 -> 1, 2 -> 2, 3 -> 3 .... 10000000 -> 10000000

  • LevelDB - 165.9 MB
  • SwayDB Map - 142.1 MB
  • SwayDB Set - 132.1 MB (Set databases reduces the space usage even further)

Unique keys only and same/duplicate values

Key -> Value = 1 -> 1, 2 -> 1, 3 -> 1 .... 10000000 -> 1

  • LevelDB - 165.9 MB
  • SwayDB Map - 92.1 MB (Because SwayDB eliminates duplicate values and writes them only ones the space usage is reduced even further)
  • SwayDB Set - 132.1 MB (Set combines key-values into a single key so there no duplicates)

I'm not too familiar with LevelDB's configuration but I used https://github.com/fusesource/leveldbjni with the following options.

val options = new Options()
options.createIfMissing(true)
options.compressionType(CompressionType.NONE)
options.writeBufferSize(10000000)

For SwayDB I used the default with none groupingStrategy which disables compression.

Peter C.
@touhonoob
@simerplaha Wow that's impressive. Thanks for your time.
Simer Plaha
@simerplaha
No worries. Let me know if anything else :)
Valentyn Kolesnikov
@javadev
Java wrapper is available in maven central repository. :+1:
Simer Plaha
@simerplaha
That's awesome Valentyn. Thank you so much.
Simer Plaha
@simerplaha
Hey @/all just created a slack account for some of us who prefer slack - here is the invite link.
algobardo
@algobardo
@simerplaha impressive work! I’m playing around with Scala, FP, monix and swaydb for self-enjoyment. I have a few questions, I would love if you find some time to answer them.
  • you say that sway is non-blocking internally, but the async API seems to only “wrap” the sync one in an asyncronous manner. Can you give me pointers to understand if what I say is true or false?
  • what is the status of the integration with monix to replace the Future with monix Tasks ?
Simer Plaha
@simerplaha

Hi @algobardo. Async APIs should not wrap sync APIs. I'm wondering if you are looking at an old version which had Async APIs work-in-progress?

In the newer versions (0.8-BETA+) all APIs are abstracted with the type Tag[T[_]] which allows us to choose whichever container we want to use for Sync and Async APIs.

For example: if you want to use Monix Task.

image.png
Same goes with Stream
image.png
As far as replacing Future with monix goes, the above approach allows us to use all container like Monix’s Task, Scala Future or Try, Scalaz Task, ZIO etc.
Simer Plaha
@simerplaha
in 0.8-BETA+ the tagAsync is named asyncAPI which is now simplified and will be in the next version.
Simer Plaha
@simerplaha
My bad I forgot. Writes do use sync API within async to maintain the order of inserts and updates. This might change depending on if there is a performance or reduced resource benefit. Will have to run some benchmarks.
Simer Plaha
@simerplaha

@algobardo hey mate, just pushed another release with Monix support. Added 2 examples here demoing how to use Task.

I've never worked with monix before so any suggestions on improving the examples or testing would help a lot.

Let me know how you go.

Ori Dagan
@oridag
Hi @simerplaha, first let me say that your project looks great. A fast, non-blocking, scala-native embedded DB is a welcome addition to the ecosystem. We are considering giving it a try so I wanted to ask about its production readiness and if you are planning a production release any time soon. Thanks!
Simer Plaha
@simerplaha

Hi @oridag, thank you. Supporting scala-native is something I'm looking forward to as well (#1).

I wish I could say that we are production ready now but we need to write more integration test-cases (#178) and finish few tasks relating to measuring and monitoring performance (specially #276).

Best case scenario is that we reach production readiness before new year otherwise we will definitely be production ready by early-mid next year.

Ori Dagan
@oridag
Got it. Thanks!
Simer Plaha
@simerplaha
:)
Ori Dagan
@oridag
Hi @simerplaha , just checking to see if there is any new ETA for production release?
Simer Plaha
@simerplaha
Hi @oridag, I haven’t been able to find much OS time recently. Can’t say much on the ETA but we are not that far from it.
Glen Marchesani
@fizzy33
hello I am wondering on cats effect 3 support ?
Simer Plaha
@simerplaha
You can copy this Bag implementation into your code. Proper release will happen when I win the battle against JVM's garbage collector.
import cats.effect.IO
import cats.effect.unsafe.IORuntime
import swaydb.Bag.Async
import swaydb.serializers.Default._
import swaydb.{IO => SwayIO, _}

import scala.concurrent.{ExecutionContext, Future, Promise}
import scala.util.Failure

object Cats3Example extends App {

  /**
   * Cats-effect 3 async bag implementation
   */
  implicit def apply(implicit runtime: IORuntime): swaydb.Bag.Async[IO] =
    new Async[IO] { self =>

      override def executionContext: ExecutionContext =
        runtime.compute

      override val unit: IO[Unit] =
        IO.unit

      override def none[A]: IO[Option[A]] =
        IO.pure(Option.empty)

      override def apply[A](a: => A): IO[A] =
        IO(a)

      override def map[A, B](a: IO[A])(f: A => B): IO[B] =
        a.map(f)

      override def transform[A, B](a: IO[A])(f: A => B): IO[B] =
        a.map(f)

      override def flatMap[A, B](fa: IO[A])(f: A => IO[B]): IO[B] =
        fa.flatMap(f)

      override def success[A](value: A): IO[A] =
        IO.pure(value)

      override def failure[A](exception: Throwable): IO[A] =
        IO.fromTry(Failure(exception))

      override def foreach[A](a: IO[A])(f: A => Unit): Unit =
        f(a.unsafeRunSync())

      def fromPromise[A](a: Promise[A]): IO[A] =
        IO.fromFuture(IO(a.future))

      override def complete[A](promise: Promise[A], a: IO[A]): Unit =
        promise tryCompleteWith a.unsafeToFuture()

      override def fromIO[E: SwayIO.ExceptionHandler, A](a: SwayIO[E, A]): IO[A] =
        IO.fromTry(a.toTry)

      override def fromFuture[A](a: Future[A]): IO[A] =
        IO.fromFuture(IO(a))

      override def suspend[B](f: => IO[B]): IO[B] =
        IO.defer(f)

      override def flatten[A](fa: IO[IO[A]]): IO[A] =
        fa.flatMap(io => io)
    }

  implicit val runtime = IORuntime.global

  val test =
    for {
      map <- memory.Map[Int, String, Nothing, IO]()
      _ <- map.put(key = 1, value = "one")
      value <- map.get(key = 1) //returns "one"
    } yield {
      println(s"value: $value")
    }

  test.unsafeRunSync()
}
Glen Marchesani
@fizzy33
thanks @simerplaha I see you did a cats effect 3 release so I am guessing you won the battle with the garbage collector :-)
cooking with gas on swaydb... I would love to understand more about why you made swaydb. It looks like the perfect kit for my project (have previously used pulsar and then rocksdb)..
the ability to expire multi-maps covers my use cases nicely. My use case is http based messaging middleware with a large number of topics / mailboxes.
where swaydb would be the underlying storage engine
I have it prototyped in and all looks well. Wondering if there are any performance nobs I should turn.
Right now it is a direct port from the RocksDB code so topic and database partitioning is not managed by the storage engine BUT with multi-maps and expirations I can easily move it all into that layer...
the RocksDB code we create a RocksDB per day and at an atomic moment each day we drop the oldest rocks db from the end of partition list and add a new one to front of the list...