Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 26 15:13

    simerplaha on master

    fix 2.12 build (compare)

  • Jan 26 14:30

    simerplaha on master

    compaction related (compare)

  • Jan 26 04:14

    simerplaha on master

    DefIO (compare)

  • Jan 25 16:17

    simerplaha on master

    added PushStrategy to Compactio… (compare)

  • Jan 25 15:42

    simerplaha on master

    Split Level assign and merge remove Compaction grouping per … (compare)

  • Jan 24 14:16

    simerplaha on master

    added task.assigner package (compare)

  • Jan 24 07:06

    simerplaha on master

    fix 2.12 build (compare)

  • Jan 24 07:00

    simerplaha on master

    fix 2.12 build (compare)

  • Jan 22 19:50

    simerplaha on master

    renaming & replaced similar ove… materialise should return Itera… (compare)

  • Jan 20 15:32

    simerplaha on master

    merge all midOverlaps (compare)

  • Jan 20 13:08

    simerplaha on master

    updateCopyBinarySearch, replace… (compare)

  • Jan 19 12:42

    simerplaha on master

    CompactionLevelTaskerSpec (compare)

  • Jan 19 06:13

    simerplaha on master

    CompactionLevelZeroTasker_flatt… CompactionLevelZeroTasker_flatt… (compare)

  • Jan 18 20:29

    simerplaha on master

    CompactionLevelZeroTasker_creat… (compare)

  • Jan 18 20:26

    simerplaha on master

    CompactionLevelZeroTasker_creat… (compare)

  • Jan 18 20:17

    simerplaha on master

    CompactionLevelZeroTasker_creat… (compare)

  • Jan 18 20:11

    simerplaha on master

    CompactionLevelZeroTasker_creat… (compare)

  • Jan 18 18:09

    simerplaha on master

    compaction.task test-cases (compare)

  • Jan 17 18:19

    simerplaha on master

    renamed to toLastLevelOrNull (compare)

  • Jan 17 18:17

    simerplaha on master

    renamed to addLastLevelOrNull (compare)

Mitchell Yuwono
@myuwono
See if I can set it up for you
Simer Plaha
@simerplaha
Yea sure. Go for it.
Frank Rosner
@FRosner
Hi! I just stumbled upon SwayDB. I was wondering why you built it and what you are using it for?
Frank Rosner
@FRosner
Also I see that there is still no travis :D Can I help? Looks like running sbt test and getting some code coverage would be a good start already?
Simer Plaha
@simerplaha

Hey Frank (@FRosner), I built it to have a fast database that is easy to use and highly scalable on a single machine.

The overall goal of the core implementation is high speeds & low storage requirement to reduce costs of hosting a set of data on single machine.

  • Use CPU & RAM efficiently to get maximum read and write throughput on a single machine without any wastage (keep the cores busy & reduce GC workload).
  • Disk - distribute data across multiple disks and to get maximum file compression.
  • API - easy to use type-safe API. Slick is a great example for us to see that with a simple collection like API we can query complex data structures (SQL databases).

The goals of the database in detail can be read on the documentation website: http://www.swaydb.io/goals/

Having Travis setup will be great help. But just a little heads up - I've added a comment to this ticket created by @myuwono outlining the issue with running test cases. You will most likely hit a FileSystemException: Too many open files when running the test cases. But having travis setup would be very helpful.

I'm not aware of anyone using SwayDB in production right now. FYI: Versions 0.3 & 0.4 will be going through major file format changes to support compression and will not be backward compatible.

Frank Rosner
@FRosner

Having Travis setup will be great help. But just a little heads up - I've added a comment to this ticket created by @myuwono outlining the issue with running test cases. You will most likely hit a FileSystemException: Too many open files when running the test cases. But having travis setup would be very helpful.

I see... Ok let me think about what we can do.

Frank Rosner
@FRosner
Do you have a .travis.yml that didn't work? Afaik you should be able to change the ulimit if you request a sudo enabled build environment. Have you tried that?
Frank Rosner
@FRosner
I have another question, @simerplaha. I was looking at your benchmark results and I noticed that if the keys are put in order the write throughput is doubled compared to shuffled keys. Why is that?
I am aware of the concept behind LSM trees but I can't really make sense from that. What is the data structure used for the individual levels?
image.png
Also why is the difference getting less when moving from 2-level in-memory towards 8-level regular file?
Simer Plaha
@simerplaha
This message was deleted
Simer Plaha
@simerplaha

Hi Frank, I haven't looked at setting up travis yet.

Random inserts are slower than sequential because of insertion cost of inserting randomly into skipList. That's just how skipLists are. In Level0 after key-values get written to a .log file they get added to an in-memory skipList. Sequential insertion would just append to the end of the skipList and adding new upper Levels (in the skipList) as required, but random insertions result in modifying and re-linking (similar to linkedList) multiple keys in the skipList and sometimes multiple Levels (skipList hierarchies).

You can run a simple test with java.util.concurrent.ConcurrentSkipListMap to see the performance difference.

By data structure I assuming you mean the 'file formats'. I don't think the current file formats are documented anywhere. The overview of what Map files (.log) and Segment files (.seg) are can be found on the website. The actual file formats needs to be documented. Have a look at the documentation at http://www.swaydb.io/terminology/ which is an overview of each file and the role they play in Levels which should give you a good understanding of how the Levels are structured.

Thank you for the chart :+1:

Simer Plaha
@simerplaha

Also why is the difference getting less when moving from 2-level in-memory towards 8-level regular file?

As you know RAM is a lot faster than MMAP files and MMAP files are faster than regular files (java.io.FileChannel).

The benchmarks should also show performance results when the databases cache is partly populated. As you read key-values they get cached in-memory and performance increases drastically. Reads get a lot faster as the database is running because the most read keys are in-memory which other reads piggy back off to save disk seeks.

Simer Plaha
@simerplaha
Please let me know if things are unclear.
Frank Rosner
@FRosner

As you know RAM is a lot faster than MMAP files and MMAP files are faster than regular files (java.io.FileChannel).

This is clear to me, of course. I was just wondering why the relative performance penalty becomes less for regular files than for MMAP files / in-memory. Do you know what I mean?

I now also understand that you are using skip lists as data structures for the in-memory levels. What are you using for storing on disk? B trees? SSTables? Some other structure?

Hi Frank, I haven't looked at setting up travis yet.

Ok I will see if I can find the time. What should travis run? sbt test?

Thank you for the chart :thumbsup:

I am currently writing a blog post series about data structures to refresh my knowledge. Is it ok if I use your numbers to visualize the effects of compaction and different persistence levels on the throughput? That's where the graph was going to go. Here is the latest post of the series: Read Efficient Data Structures. I am currently working on Update Efficient Data Structures.

Simer Plaha
@simerplaha

What are you using for storing on disk? B trees? SSTables? Some other structure?

I think its similar to SSTables. Here is an overview of the format - the Segment files store bytes in 3 groups.
1- Values - Top of the files stores all values bytes.
2 - Index - Keys stored in sorted order with other metadata related to the key such as id, compression info, value offset, TTL etc.
3 - Footer - stores bloomfilter & other file format information like key-value count, CRC, index offsets.

Ok I will see if I can find the time. What should travis run? sbt test?

Yep sbt test is correct.

Is it ok if I use your numbers to visualize the effects of compaction and different persistence levels on the throughput?

Yes of course that's OK. Looking forward to your blog post :+1:

... Here is the latest post of the series: Read Efficient Data Structures. I am currently working on Update Efficient Data Structures.

I had a quick read and it looks very interesting. I will have a proper read soon.

Simer Plaha
@simerplaha

The reason I used a custom format to get better compression.
A test inserting 10000000 key-values with compression and MMAP disabled resulted in following disk usage.

  • LevelDB - 200.3 mb
  • RocksDB - 197.6 mb
  • SwayDB - 149.5 mb

The above is just a quick test. I might be missing something from my test to tune LevelDB & RocksDB. It will be interesting to see the disk usage after releasing version 0.4 which will have better compressed file format and also support for LZ4.

Frank Rosner
@FRosner
Great! Thanks for sharing this awesome project!
Simer Plaha
@simerplaha
No worries :+1:
Simer Plaha
@simerplaha
Thank you for your awesome feedback :- )
Simer Plaha
@simerplaha

@/all Hey guys just released version 0.3. This release contains support for expire API (TTL), update & improved batching for atomic writes. Here is a sample of some of the new API.

//put & expire a key-value after a day
db.put(key = 1, value = "one", expireAfter = 1.day)
//or expire a range of key-values after an hour
db.expire(from = 1, to = 1000, after = 1.hour)
//update values without altering the already set expiration
db.update(from = 1, to = 1000, value = "value updated")
//or update a single value
db.update(key = 1, value = "value updated")
//fetch the expiration deadline for a key
db.expiration(key = 1)
//fetch time left until the key's expiration
db.timeLeft(key = 1)

Here is a whole list of write AP and read API.

I'm going to start working on getting compression ready for v0.4. If you think of any features you think we should add please do bring it up.

Other than that, all the core APIs are done. Any new features can be built as extension libraries. Let me know what you think.
Frank Rosner
@FRosner
I don't really have any strong opinions but great work releasing 0.3!
Also I couldn't find the time to look into the travis build :(
Simer Plaha
@simerplaha

Thank Frank.
No worries whenever you have time is OK. We will eventually get to it :)

Loved your blog posts BTW :+1:

Simer Plaha
@simerplaha

@/all Just pushed a big commit to support compression. Both LZ4 and Snappy libraries are full supported. It's part of the grouping strategy you can read up here .

Segment file format documentation
Grouping format documentation

Will do a release after simerplaha/SwayDB#15

Frank Rosner
@FRosner
Yeah :thumbsup:
Simer Plaha
@simerplaha
@/all Just released v0.4. I've added a topic to scala-lang with a brief outlining the release.
Peter Chung
@touhonoob
Hello There, Do you guys have any stats on the storage footprint of SwayDB in comparison to LevelDB or LMDB?
in terms of mmap file
Simer Plaha
@simerplaha

Hi @touhonoob, just ran some quick space usages tests on 10 million key-values with compression disabled for both LevelDB and SwayDB.

SwayDB provides two storage formats Map and Set. So I've added space usage for both.

The following shows the total size of sstables created by LevelDB and segments files by SwayDB.

Space usage when keys and values are unique integers

Key -> Value = 1 -> 1, 2 -> 2, 3 -> 3 .... 10000000 -> 10000000

  • LevelDB - 165.9 MB
  • SwayDB Map - 142.1 MB
  • SwayDB Set - 132.1 MB (Set databases reduces the space usage even further)

Unique keys only and same/duplicate values

Key -> Value = 1 -> 1, 2 -> 1, 3 -> 1 .... 10000000 -> 1

  • LevelDB - 165.9 MB
  • SwayDB Map - 92.1 MB (Because SwayDB eliminates duplicate values and writes them only ones the space usage is reduced even further)
  • SwayDB Set - 132.1 MB (Set combines key-values into a single key so there no duplicates)

I'm not too familiar with LevelDB's configuration but I used https://github.com/fusesource/leveldbjni with the following options.

val options = new Options()
options.createIfMissing(true)
options.compressionType(CompressionType.NONE)
options.writeBufferSize(10000000)

For SwayDB I used the default with none groupingStrategy which disables compression.

Peter Chung
@touhonoob
@simerplaha Wow that's impressive. Thanks for your time.
Simer Plaha
@simerplaha
No worries. Let me know if anything else :)
Valentyn Kolesnikov
@javadev
Java wrapper is available in maven central repository. :+1:
Simer Plaha
@simerplaha
That's awesome Valentyn. Thank you so much.
Simer Plaha
@simerplaha
Hey @/all just created a slack account for some of us who prefer slack - here is the invite link.
algobardo
@algobardo
@simerplaha impressive work! I’m playing around with Scala, FP, monix and swaydb for self-enjoyment. I have a few questions, I would love if you find some time to answer them.
  • you say that sway is non-blocking internally, but the async API seems to only “wrap” the sync one in an asyncronous manner. Can you give me pointers to understand if what I say is true or false?
  • what is the status of the integration with monix to replace the Future with monix Tasks ?
Simer Plaha
@simerplaha

Hi @algobardo. Async APIs should not wrap sync APIs. I'm wondering if you are looking at an old version which had Async APIs work-in-progress?

In the newer versions (0.8-BETA+) all APIs are abstracted with the type Tag[T[_]] which allows us to choose whichever container we want to use for Sync and Async APIs.

For example: if you want to use Monix Task.

image.png
Same goes with Stream
image.png
As far as replacing Future with monix goes, the above approach allows us to use all container like Monix’s Task, Scala Future or Try, Scalaz Task, ZIO etc.
Simer Plaha
@simerplaha
in 0.8-BETA+ the tagAsync is named asyncAPI which is now simplified and will be in the next version.
Simer Plaha
@simerplaha
My bad I forgot. Writes do use sync API within async to maintain the order of inserts and updates. This might change depending on if there is a performance or reduced resource benefit. Will have to run some benchmarks.
Simer Plaha
@simerplaha

@algobardo hey mate, just pushed another release with Monix support. Added 2 examples here demoing how to use Task.

I've never worked with monix before so any suggestions on improving the examples or testing would help a lot.

Let me know how you go.

Ori Dagan
@oridag
Hi @simerplaha, first let me say that your project looks great. A fast, non-blocking, scala-native embedded DB is a welcome addition to the ecosystem. We are considering giving it a try so I wanted to ask about its production readiness and if you are planning a production release any time soon. Thanks!
Simer Plaha
@simerplaha

Hi @oridag, thank you. Supporting scala-native is something I'm looking forward to as well (#1).

I wish I could say that we are production ready now but we need to write more integration test-cases (#178) and finish few tasks relating to measuring and monitoring performance (specially #276).

Best case scenario is that we reach production readiness before new year otherwise we will definitely be production ready by early-mid next year.

Ori Dagan
@oridag
Got it. Thanks!
Simer Plaha
@simerplaha
:)