Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

yevhen/Streamstone

Simple event store for Table Storage

People

Repo info

Activity

Mar 22 17:29

yevhen commented #61
Mar 19 09:30

iamim commented #61
Mar 19 08:50

yevhen commented #61
Mar 19 08:50

yevhen commented #61
Mar 18 12:25

iamim commented #61
Mar 15 10:49

yevhen commented #61
Mar 15 10:46

yevhen commented #61
Mar 15 10:46

yevhen commented #61
Mar 15 10:25

yevhen commented #61
Mar 09 12:25

iamim commented #61
Mar 01 10:06

iamim closed #63
Mar 01 10:06

iamim commented #63
Mar 01 10:02

yevhen commented #63
Mar 01 09:28

iamim opened #63
Feb 24 15:01

colgreen commented #54
Feb 24 11:17

yevhen commented #54
Feb 23 19:28

colgreen commented #54
Feb 16 15:26

seniorquico commented #62
Feb 16 14:01

yevhen closed #62
Feb 16 14:01

yevhen commented #62

Yevhen Bobrov

@yevhen

I think @jkonecki might have some expertise running SS with CosmosDB

@lufthavn

What kind of problem you

have with it

Tobias Kastrup Andersen

@lufthavn

Well, right now I simply don't know how to connect my app to the Cosmos emulator. Can't seem to find a valid connection string.

Yevhen Bobrov

@yevhen

2.3.0 on nuget. Updated to work with latest Azure storage library. Starting from 9.4 the library was merged with Cosmos api and namespaces were changed. Kudos to @ahocquet for updating SS!

Elan Hasson

@ElanHasson

Hi Yevhen

I've been looking at Streamstone and have a few questions

1- Snapshots-- are they conceptually the same as projections of the events?

Elan Hasson

@ElanHasson

2- I've spent a great deal of time reading the code, it looks like this is event store as a library vs an event store as a database. Is this a good understanding of it?

Yevhen Bobrov

@yevhen

@ElanHasson yes, it cleary states that in the tagline https://github.com/yevhen/Streamstone/blob/master/README.md

1-Snapshots. Yes, it's just another projection which has 1-1 correspondence with in-memory representation of aggregate's state. just an optimization if you don't want to replay all events every time you bring an aggregate (backed by actor) into memory

Elan Hasson

@ElanHasson

Thanks. Does streamstone

Have support for retrieving the snapshot? Or is that left to user?

Yevhen Bobrov

@yevhen

@ElanHasson yes. both reading and storing is completely up to the user. For storing you may use: 1) inline projections (ETG) for snapshots which are fully consistent with latest event; 2) saving snapshots out-of-band (for example by using a timer)

Alex Gordon

@alexgman

hey guys! could someone give me an example of how i would do projections over table storage, for eaxmple lets say i have a bunch events in my table, how do i create aggregate queries?

Elan Hasson

@ElanHasson

@alexgman Check S10_Stream_directory

That may give you an idea on how you could iterate through all streams.

for a single stream, see S05_Read_from_stream

As far as creating projections for aggregate snapshots, I'm looking at https://liquidprojections.net to help automate that and make it easier.

Elan Hasson

@ElanHasson

You may also want to check out S06_Include_additional_entities, which includes a demo on how to include snapshots in the stream

Hi @yevhen, I'm persisting an event with a string Id to support dupe detection using my entity's natural key.

From SS perspective, How would I handle if the user wants to rename the property which backs the key?

Option 1: Keys are for life, do not support renaming of the natural key
Option 2: Update the row key for all objects in the stream (headache?)

Elan Hasson

@ElanHasson

@alexgman yevhen/Streamstone#44

Yevhen Bobrov

@yevhen

got custom domain name back http://streamstone.net/ Thanks to the @ElanHasson for donation!!!

Matej Hertis

@mhertis

Special question, what can one do if Stream headers was manually changed in storage and now StreamStone is throwing `Streamstone.ConcurrencyConflictException at Streamstone.Stream+WriteOperation+Batch.HandleStream header has been changed or already exists in a storage :)

Yevhen Bobrov

@yevhen

Reload)

If stream is in memory the only way you can fix the situation is to reload it from storage to sync the etag

Matej Hertis

@mhertis

ok, thnx :)

was afraid of that answer :)

Robert Friberg

@rofr

I'm writing a storage backend for Memstate based on StreamStone, feels really solid so far. Nice work 👊

Yevhen Bobrov

@yevhen

@rofr thanks! If you need any help - don't hesitate to use this chat)

Robert Friberg

@rofr

@rofr thanks! If you need any help - don't hesitate to use this chat)

Thanks, good to know. I have a somewhat working version now:
https://github.com/DevrexLabs/memstate/tree/v2/src/Memstate.Azure

Robert Friberg

@rofr

Feel free to take a peek, any feedback would be greatly appreciated. BTW, what kind of throughput might I expect to see?

Yevhen Bobrov

@yevhen

@rofr ic what you're doing there) looks good

@rofr it's hard to predict throughput since it will depend on multiple factors, like batch size (volume) and how frequent are concurrent writes, as you don't have a single writer

Yevhen Bobrov

@yevhen

the hard cap for partition is 2K entities per second but it's for non-concurrent parallel writes. With event streams, you write sequentially due to optimistic concurrency (the writes are basically serialized) so it's different story

you might get 200-300 events per second (2-3 batches x100 written sequentially) if you're lucky with write latency and event arrival (ie you can form full batch very quickly).

Yevhen Bobrov

@yevhen

in case of concurrency when you write from multiple machines into the same stream - it will degrade considerably due to re-reads

so you can improve throughput by employing a single writer utilizing some sort of distributed application framework like Akka or Orleans. Actors are great hosts for streams

Orleans might give a boost to memstate by providing a way to distribute state - both sharding and replication, without sacrificing availability

Robert Friberg

@rofr

Thanks for the feedback @yevhen, I appreciate it

Robert Friberg

@rofr

I would classify memstate as a replicated state machine favoring consistency (CP) over availability (AP). Normally the entire system is a single (huge) event-sourced aggregate.

Robert Friberg

@rofr

OrigoDB, predecessor of memstate, has a sharding/partitioning but we haven't ported it yet as there is not much demand and there are other features with higher priority.

Robert Friberg

@rofr

so you can improve throughput by employing a single writer utilizing some sort of distributed application framework like Akka or Orleans. Actors are great hosts for streams

OrigoDB has a primary + read replica(s) architecture but no leader election, so failover requires manual intervention. But I have an idea about how to do leader election using commands embedded in the event stream allowing a primary + replica model for memstate as well.

Have you thought about supporting DynamoDB? I think the same schema and concepts can be applied there as well!

And how about cosmosdb?

cc @goblinfactory

Yevhen Bobrov

@yevhen

For CosmosDB check the Eveneum project from @jkonecki

I have no plans for DynamoDB at the moment

Chat via Matrix