Some abstraction to manage a group of FASTER instances would be interesting, yes. It's also useful to see if you can somehow use a single FASTER instance, but disambiguate records by a prefix in the key. Doesn't apply in all scenarios, of course, but worth checking out.
This whole thing is bigger than I'm able to really wrap my brain around, so I keep trying to manage little bites at a time... If any of you here have any time and would like to be a part of this research/experiment/project, I could use all the braincells you can muster!
well blimey. the abstraction i created to combine those caches now has opened up some more opportunities.. now I'm debating going down to a single cache.
🤔 I love a good challenge
Stefan
@Tornhoof
Badrish, are you planning to release a new build on nuget?
Badrish Chandramouli
@badrishc
@Tornhoof - yes, we will release a new build this week.
Badrish Chandramouli
@badrishc
v1.9.6 was just put out.
Stefan
@Tornhoof
Thank you very much.
Chris Ochs
@gamemachine
Didn't see anything on this in the docs. Is ClientSession reused internally? Or some other way to avoid creating a new object every use.
Ah I see the reference to a session pool, so assuming it's safe to just have my own wrappers that keep a session alive for some time.
Having some difficulty connecting the dots on exactly how you can end up data only on disk for the KV store,where a read would return pending. Or is that scenario only applicable to logs?
Badrish Chandramouli
@badrishc
If you instantiate a FASTER KV with small memory size (via LogSettings), it will spill the older data to disk. Also, calling store.Log.FlushAndEvict(true) will force an eviction of all in-memory records to disk. After this, any Read operation will always return PENDING as it needs to issue a disk IO to retrieve the record from disk. Or you can use the async-based ReadAsync call.
Chris Ochs
@gamemachine
Ah ok makes perfect sense thanks
Masashi Ito
@mito-csod
I see mentions of "record expiration" in the description for the key iterator, but is record expiration a feature (like, the kind based on an expiry date), or is that simply referring to entries which may have been evicted from memory and subsequently truncated out of the log?
Badrish Chandramouli
@badrishc
Key iteration does an iteration of all live keys, i.e. keys that are not explicitly deleted or truncated from the begin of the log. One uses ShiftBeginAddress to perform expiration of older records in the hybrid-log, in bulk from the begin of the log.
Masashi Ito
@mito-csod
Thanks. Is there some clever way to do a date-based expiry of a record by key? I could iterate through all keys, check the some expiration date value, and then delete the key, but I feel like there'd be a more efficient way that could exploit the linear nature of the writes, especially if the expiration dates are immutable and based only on the creation time.
Badrish Chandramouli
@badrishc
Two thoughts:
ShiftBeginAddress does eliminate the oldest records, so it approximates date based expiry.
store.Log.Compact takes a compaction function as input. You can provide expiration condition here. Only the unexpired records will be copied over to tail. Then the compacted region can be shifted out (deleted).
Badrish Chandramouli
@badrishc
Or do you want to only delete a particular key? In this case, do a random Read, check the date, then do a Delete operation, which will only write a tombstone record to the tail so will be efficient.
3 replies
Masashi Ito
@mito-csod
Am I doing something wrong if the output of an IObjectSerializer is getting written into the object log and not the main log? I had imagined that it would go into the main log like Memory<T>
7 replies
Chris Ochs
@gamemachine
So tested running faster in Unity3D and found some strange behaviour. The log isn't getting created on disk. Exact same code (shared dll) works fine in .net 5. probably as a result of that checkpoint returns true on the first try and then fails after that.
1 reply
Jorge Candeias
@JorgeCandeias
@badrishc Hi Badrish. This is off-topic but I've noticed the Trill repository hasn't been updated in about six months. Is Trill being discontinued or replaced with something else?
2 replies
roeeiit1
@roeeiit1
Hi, I wanted to know if faster can fit my use case, I'm searching for a way to keep a persistent state for a scaled microservice with multiple instances via a shared storage The state is mostly write only and not a lot of updates but high reads
1 reply
AmirEhsan
@iamirehsan
@badrishc hi bradrish , i have an stupid question but my code does not write anything in log file that I have when I upsert something and then I can not read from it and it return 0 in output I really need help
AmirEhsan
@iamirehsan
@ralphbecket hey man i have a same problem that you had any advise?
Masashi Ito
@mito-csod
Is there a way to wire up a callback to a deletion event while having access to the value that is getting deleted? the callbacks in IFunctions and IAdvancedFunctions only has the key, presumably because this is triggering after the data is really gone.
Badrish Chandramouli
@badrishc
Deletion can be blind, where the older value is on disk and hence unknown. So in general it can't be done with the Delete API. But in v2, it can be done using RMW's new Expiration capability. You would set NeedInitialUpdate to false, and both InPlaceUpdater and CopyUpdater will set rmwInfo.Action to either RMWAction.ExpireAndStop (if "rollover" logic is not required) or to RMWAction.ExpireAndResume (if "rollover" logic is required), You can then hook your deletion callback into InPlaceUpdater and PostCopyUpdater.
Masashi Ito
@mito-csod
..and i see 2.0 is released. Will check it out. Thank you :)
Masashi Ito
@mito-csod
@badrishc did the behavior of record deletion through compaction change in 2.0? i call compact() from one session that deletes a record (i had only 1 record as a test, and i see the record count drop to 0 after this), but then another session looking up the deleted record by key still gets the record back. in 1.9.x, the same code would not find the record (which is what i expected).
17 replies
Masashi Ito
@mito-csod
is fasterkv expected to work with unsafe blittable structs with fixed arrays inside (like a fixed char[32])?