Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    at the moment I've got a weird mix of type safety/fixed members and dynamic collections
    shrug this whole thing is a research experiment to see how it ends up shaping.
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    Some kind of more generic shared session object like im creating could potentially be useful to others
    goal is to keep several different types of faster stores at least checkpointed at the same places/events
    Badrish Chandramouli
    @badrishc
    Some abstraction to manage a group of FASTER instances would be interesting, yes. It's also useful to see if you can somehow use a single FASTER instance, but disambiguate records by a prefix in the key. Doesn't apply in all scenarios, of course, but worth checking out.
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    Thank you again on both counts
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    This whole thing is bigger than I'm able to really wrap my brain around, so I keep trying to manage little bites at a time... If any of you here have any time and would like to be a part of this research/experiment/project, I could use all the braincells you can muster!
    anyone!
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    how did you get your team, Badrish?
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    How do you find qualified people with interest and time?
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    I don't suppose you have any time to look how I'm starting to implement my session/commit in the Faster/CacheManager/FasterBlockCacheManager.* files?
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    working on reducing number of KVs and having a multi-purpose store like you mentioned
    I think I can combine 2 or 3 KVs I was going to keep separate if I can get the right hand side class structure right :)
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    Hey, it seems my index combining actually makes a huge performance difference. My test suite is 12-20s faster.
    cool!
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    I've reduced it by about 50s in total.
    There's a heck of a lot of setup and teardown of all the KVs in my tests, so it really multiplies.
    Jessica Mulein
    @jessicamulein:matrix.org
    [m]
    well blimey. the abstraction i created to combine those caches now has opened up some more opportunities.. now I'm debating going down to a single cache.
    🤔 I love a good challenge
    Stefan
    @Tornhoof
    Badrish, are you planning to release a new build on nuget?
    Badrish Chandramouli
    @badrishc
    @Tornhoof - yes, we will release a new build this week.
    Badrish Chandramouli
    @badrishc
    v1.9.6 was just put out.
    Stefan
    @Tornhoof
    Thank you very much.
    Chris Ochs
    @gamemachine
    Didn't see anything on this in the docs. Is ClientSession reused internally? Or some other way to avoid creating a new object every use.
    Ah I see the reference to a session pool, so assuming it's safe to just have my own wrappers that keep a session alive for some time.
    Badrish Chandramouli
    @badrishc
    ClientSession is meant to be allocated once and reused for a sequence of mono-threaded interactions with the store. You can pool them as we do in the sample at https://github.com/microsoft/FASTER/blob/master/cs/playground/AsyncStress/SerializedFasterWrapper.cs
    Chris Ochs
    @gamemachine
    Having some difficulty connecting the dots on exactly how you can end up data only on disk for the KV store,where a read would return pending. Or is that scenario only applicable to logs?
    Badrish Chandramouli
    @badrishc
    If you instantiate a FASTER KV with small memory size (via LogSettings), it will spill the older data to disk. Also, calling store.Log.FlushAndEvict(true) will force an eviction of all in-memory records to disk. After this, any Read operation will always return PENDING as it needs to issue a disk IO to retrieve the record from disk. Or you can use the async-based ReadAsync call.
    Chris Ochs
    @gamemachine
    Ah ok makes perfect sense thanks
    Masashi Ito
    @mito-csod
    I see mentions of "record expiration" in the description for the key iterator, but is record expiration a feature (like, the kind based on an expiry date), or is that simply referring to entries which may have been evicted from memory and subsequently truncated out of the log?
    Badrish Chandramouli
    @badrishc
    Key iteration does an iteration of all live keys, i.e. keys that are not explicitly deleted or truncated from the begin of the log. One uses ShiftBeginAddress to perform expiration of older records in the hybrid-log, in bulk from the begin of the log.
    Masashi Ito
    @mito-csod
    Thanks. Is there some clever way to do a date-based expiry of a record by key? I could iterate through all keys, check the some expiration date value, and then delete the key, but I feel like there'd be a more efficient way that could exploit the linear nature of the writes, especially if the expiration dates are immutable and based only on the creation time.
    Badrish Chandramouli
    @badrishc
    Two thoughts:
    • ShiftBeginAddress does eliminate the oldest records, so it approximates date based expiry.
    • store.Log.Compact takes a compaction function as input. You can provide expiration condition here. Only the unexpired records will be copied over to tail. Then the compacted region can be shifted out (deleted).
    Badrish Chandramouli
    @badrishc
    Or do you want to only delete a particular key? In this case, do a random Read, check the date, then do a Delete operation, which will only write a tombstone record to the tail so will be efficient.
    3 replies
    Masashi Ito
    @mito-csod
    Am I doing something wrong if the output of an IObjectSerializer is getting written into the object log and not the main log? I had imagined that it would go into the main log like Memory<T>
    7 replies
    Chris Ochs
    @gamemachine
    So tested running faster in Unity3D and found some strange behaviour. The log isn't getting created on disk. Exact same code (shared dll) works fine in .net 5. probably as a result of that checkpoint returns true on the first try and then fails after that.
    1 reply
    Jorge Candeias
    @JorgeCandeias
    @badrishc Hi Badrish. This is off-topic but I've noticed the Trill repository hasn't been updated in about six months. Is Trill being discontinued or replaced with something else?
    2 replies
    roeeiit1
    @roeeiit1
    Hi, I wanted to know if faster can fit my use case, I'm searching for a way to keep a persistent state for a scaled microservice with multiple instances via a shared storage
    The state is mostly write only and not a lot of updates but high reads
    1 reply
    AmirEhsan
    @iamirehsan
    @badrishc hi bradrish , i have an stupid question but my code does not write anything in log file that I have when I upsert something and then I can not read from it and it return 0 in output I really need help
    AmirEhsan
    @iamirehsan
    @ralphbecket hey man i have a same problem that you had any advise?
    Masashi Ito
    @mito-csod
    Is there a way to wire up a callback to a deletion event while having access to the value that is getting deleted? the callbacks in IFunctions and IAdvancedFunctions only has the key, presumably because this is triggering after the data is really gone.
    Badrish Chandramouli
    @badrishc
    Deletion can be blind, where the older value is on disk and hence unknown. So in general it can't be done with the Delete API. But in v2, it can be done using RMW's new Expiration capability. You would set NeedInitialUpdate to false, and both InPlaceUpdater and CopyUpdater will set rmwInfo.Action to either RMWAction.ExpireAndStop (if "rollover" logic is not required) or to RMWAction.ExpireAndResume (if "rollover" logic is required), You can then hook your deletion callback into InPlaceUpdater and PostCopyUpdater.
    Masashi Ito
    @mito-csod
    ..and i see 2.0 is released. Will check it out. Thank you :)
    Masashi Ito
    @mito-csod
    @badrishc did the behavior of record deletion through compaction change in 2.0? i call compact() from one session that deletes a record (i had only 1 record as a test, and i see the record count drop to 0 after this), but then another session looking up the deleted record by key still gets the record back. in 1.9.x, the same code would not find the record (which is what i expected).
    17 replies
    Masashi Ito
    @mito-csod
    is fasterkv expected to work with unsafe blittable structs with fixed arrays inside (like a fixed char[32])?
    3 replies