FASTER is a fast concurrent log and key-value store library in C# and C++. Learn more at https://github.com/microsoft/FASTER
at the moment I've got a weird mix of type safety/fixed members and dynamic collections
shrug this whole thing is a research experiment to see how it ends up shaping.
Some kind of more generic shared session object like im creating could potentially be useful to others
goal is to keep several different types of faster stores at least checkpointed at the same places/events
Not sure if this is relevant, but we have a session pool here: https://github.com/microsoft/FASTER/blob/master/cs/playground/AsyncStress/SerializedFasterWrapper.cs
Some abstraction to manage a group of FASTER instances would be interesting, yes. It's also useful to see if you can somehow use a single FASTER instance, but disambiguate records by a prefix in the key. Doesn't apply in all scenarios, of course, but worth checking out.
This whole thing is bigger than I'm able to really wrap my brain around, so I keep trying to manage little bites at a time... If any of you here have any time and would like to be a part of this research/experiment/project, I could use all the braincells you can muster!
anyone!
How do you find qualified people with interest and time?
I don't suppose you have any time to look how I'm starting to implement my session/commit in the Faster/CacheManager/FasterBlockCacheManager.* files?
working on reducing number of KVs and having a multi-purpose store like you mentioned
I think I can combine 2 or 3 KVs I was going to keep separate if I can get the right hand side class structure right :)
Hey, it seems my index combining actually makes a huge performance difference. My test suite is 12-20s faster.
cool!
There's a heck of a lot of setup and teardown of all the KVs in my tests, so it really multiplies.
well blimey. the abstraction i created to combine those caches now has opened up some more opportunities.. now I'm debating going down to a single cache.
🤔 I love a good challenge
Ah I see the reference to a session pool, so assuming it's safe to just have my own wrappers that keep a session alive for some time.
ClientSession is meant to be allocated once and reused for a sequence of mono-threaded interactions with the store. You can pool them as we do in the sample at https://github.com/microsoft/FASTER/blob/master/cs/playground/AsyncStress/SerializedFasterWrapper.cs
If you instantiate a FASTER KV with small memory size (via LogSettings), it will spill the older data to disk. Also, calling store.Log.FlushAndEvict(true) will force an eviction of all in-memory records to disk. After this, any Read operation will always return PENDING as it needs to issue a disk IO to retrieve the record from disk. Or you can use the async-based ReadAsync call.
I see mentions of "record expiration" in the description for the key iterator, but is record expiration a feature (like, the kind based on an expiry date), or is that simply referring to entries which may have been evicted from memory and subsequently truncated out of the log?
Thanks. Is there some clever way to do a date-based expiry of a record by key? I could iterate through all keys, check the some expiration date value, and then delete the key, but I feel like there'd be a more efficient way that could exploit the linear nature of the writes, especially if the expiration dates are immutable and based only on the creation time.
- ShiftBeginAddress does eliminate the oldest records, so it approximates date based expiry.
- store.Log.Compact takes a compaction function as input. You can provide expiration condition here. Only the unexpired records will be copied over to tail. Then the compacted region can be shifted out (deleted).
Deletion can be blind, where the older value is on disk and hence unknown. So in general it can't be done with the Delete API. But in v2, it can be done using RMW's new Expiration capability. You would set NeedInitialUpdate to false, and both InPlaceUpdater and CopyUpdater will set rmwInfo.Action to either RMWAction.ExpireAndStop (if "rollover" logic is not required) or to RMWAction.ExpireAndResume (if "rollover" logic is required), You can then hook your deletion callback into InPlaceUpdater and PostCopyUpdater.
@badrishc did the behavior of record deletion through compaction change in 2.0? i call compact() from one session that deletes a record (i had only 1 record as a test, and i see the record count drop to 0 after this), but then another session looking up the deleted record by key still gets the record back. in 1.9.x, the same code would not find the record (which is what i expected).
17 replies