Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

yevhen/Streamstone

Simple event store for Table Storage

People

Repo info

Activity

Mar 06 12:55

yevhen on master

Fuck Russia! 🖕 (compare)
Feb 21 11:27

yevhen on master

Fix uploading (compare)
Feb 21 11:16

yevhen on master

Add nake script for bash Fix endpoint (compare)
Feb 21 11:07

yevhen on master

Fix yml (compare)
Feb 21 10:59

yevhen on master

Restore nake first (compare)
Feb 21 10:57

yevhen on master

Fix nake command (compare)
Feb 21 10:54

yevhen on master

Fix forking (compare)
Feb 21 10:46

yevhen on master

Fork azurite process (compare)
Feb 21 10:41

yevhen on master

Build azurite (compare)
Feb 21 10:40

yevhen on master

Init submodule (compare)
Feb 21 10:33

yevhen on master

Switch to Azurite (compare)
Feb 21 10:28

yevhen on master

Add azurite as submodule (compare)
Feb 21 09:42

yevhen on master

Update build script to Nake V3 (compare)
Feb 21 09:29

yevhen on master

Extend gitignore Upgrade to .net6 Update to latest Nake and VS im… (compare)
Mar 22 2021 17:29

yevhen commented #61
Mar 19 2021 09:30

iamim commented #61
Mar 19 2021 08:50

yevhen commented #61
Mar 19 2021 08:50

yevhen commented #61
Mar 18 2021 12:25

iamim commented #61
Mar 15 2021 10:49

yevhen commented #61

Scott Ranger

@scottrangerio

Hey,

Thanks for the project :) I have a small question about virtual partitions. I'm just trying to understand what the use case / requirement is for virtual partitions. My guess would be that by using virtual partitions, we can write to multiple streams (so long as they're in the same table partition) as part of a single entity group transaction, meaning that if any of the stream write fails we have transactional behaviour. Or maybe it's simply so that multiple streams can be served from the same partition server? Are either of these assumption close to the mark or are there any different reasons / use cases for virtual partitions?

Thanks!

Yevhen Bobrov

@yevhen

@scottrangerio Hi! Yes, you've got it right! With virtual partitions you may model you whole domain as a single partition while having virtual stream for every aggregate, and you can use ETG. For example you may create stream directory in the same partition and have all operations to be atomic. You may also model simple projections as rows and update them atomically with streams. Some simple apps could be modeled in this way. Think multi-tenant setup where each tenant is a single partition. 2000K entities/sec for partition is more than enough for a majority of event-sourced apps. It is possible to develop simple invoicing multi-tenant app by just using table storage features.

Further optimization is possible by employing actor framework such as Orleans and then batch all write operations to a single partition. It is possible to go very far with this simple technology mix (Streamstone + Orleankka). Add durable sagas (you may try Orleankka's switchable behaviors for this) - and you can develop fairly complex app by just using table storage and some added cleverness :smile:

Scott Ranger

@scottrangerio

Awesome! thanks for the detailed and in-depth answer :)

ardumez

@ardumez

Do you know how to backup on CosmoDb API Table or Azure storage Table ?

Jakub Konecki

@jkonecki

Take a look at AzCopy. Not perfect but will allow you to copy the whole table.

ardumez

@ardumez

Ok thanks but there are nothing on the Azure Portal without need code ?

ardumez

@ardumez

There are backup for CosmoDb each 4h that is possible to ask on Microsoft Support on ticket

But I don't know if the CosmoDb API Table is backup with or only DocumentDB Json is backup

Jakub Konecki

@jkonecki

Azure storage geo replication gives you copies in case of hardware failure. You only need backup in order you delete something yourself that you shouldn't. I don't think there is any feature for this in Azure Portal.

Jakub Konecki

@jkonecki

Hi, are there any breaking changes in 2.x vs 1.x? I've upgraded and everything compiles / tests OK.

Yevhen Bobrov

@yevhen

yes, sync api methods were removed (due to removal in underlying Azure SDK)

Jakub Konecki

@jkonecki

Cool, I wasn't using them anyway ;-)

Yevhen Bobrov

@yevhen

so you should be fine :)

Yevhen Bobrov

@yevhen

pushed v2.1.0. Added option to disable built-in entity change tracking

Yevhen Bobrov

@yevhen

it's safe to disable it during normal stream writes. It's only useful during replays when used in conjunction with inline synchronous stream projections, which is used rarely (or not used at all). Disabling it will save few object allocations. Not really that important but nice to have ...

Yevhen Bobrov

@yevhen

2.2.0: Release Notes

Now you can restore Stream header without reading it first (via Open). If you don't pass stream properties to Stream.From() method, the header will be merged instead of replaced which means it is enough to just have previous Etag and Version to restore the header.

Matej Hertis

@mhertis

What does this actually mean, what is the use case? :)

Yevhen Bobrov

@yevhen

The use-case is simple. If you know how other event stores work in regard to optimistic concurrency, they usually requires you to pass smth like ExpectedVersion when doing writes to a stream. So you can read an aggregate, remember the last stream version and then use it as expected. You can pass this version to UI and back. That don't require read-before-write like it was in SS prior to 2.2.0. This fix gives you a similar optimization by allowing you store the pair of (etag+version) and then use it for your next write. Not everyone is using actors to always hold a StreamHeader in memory and so to resurrect this object in disconnected scenario was requiring an additional read

Matej Hertis

@mhertis

Ok, thnx, now I get it :)

Matej Hertis

@mhertis

At the time, my thinking was limited to the actors context, sorry. :)

Tobias Kastrup Andersen

@lufthavn

Has anyone figured out how to use SS and the Cosmos emulator? Is it even possible? 😊

Yevhen Bobrov

@yevhen

I think @jkonecki might have some expertise running SS with CosmosDB

@lufthavn

What kind of problem you

have with it

Tobias Kastrup Andersen

@lufthavn

Well, right now I simply don't know how to connect my app to the Cosmos emulator. Can't seem to find a valid connection string.

Yevhen Bobrov

@yevhen

2.3.0 on nuget. Updated to work with latest Azure storage library. Starting from 9.4 the library was merged with Cosmos api and namespaces were changed. Kudos to @ahocquet for updating SS!

Elan Hasson

@ElanHasson

Hi Yevhen

I've been looking at Streamstone and have a few questions

1- Snapshots-- are they conceptually the same as projections of the events?

Elan Hasson

@ElanHasson

2- I've spent a great deal of time reading the code, it looks like this is event store as a library vs an event store as a database. Is this a good understanding of it?

Yevhen Bobrov

@yevhen

@ElanHasson yes, it cleary states that in the tagline https://github.com/yevhen/Streamstone/blob/master/README.md

1-Snapshots. Yes, it's just another projection which has 1-1 correspondence with in-memory representation of aggregate's state. just an optimization if you don't want to replay all events every time you bring an aggregate (backed by actor) into memory

Elan Hasson

@ElanHasson

Thanks. Does streamstone

Have support for retrieving the snapshot? Or is that left to user?

Yevhen Bobrov

@yevhen

@ElanHasson yes. both reading and storing is completely up to the user. For storing you may use: 1) inline projections (ETG) for snapshots which are fully consistent with latest event; 2) saving snapshots out-of-band (for example by using a timer)

Alex Gordon

@alexgman

hey guys! could someone give me an example of how i would do projections over table storage, for eaxmple lets say i have a bunch events in my table, how do i create aggregate queries?

Elan Hasson

@ElanHasson

@alexgman Check S10_Stream_directory

That may give you an idea on how you could iterate through all streams.

for a single stream, see S05_Read_from_stream

As far as creating projections for aggregate snapshots, I'm looking at https://liquidprojections.net to help automate that and make it easier.

Elan Hasson

@ElanHasson

You may also want to check out S06_Include_additional_entities, which includes a demo on how to include snapshots in the stream

Hi @yevhen, I'm persisting an event with a string Id to support dupe detection using my entity's natural key.

From SS perspective, How would I handle if the user wants to rename the property which backs the key?

Option 1: Keys are for life, do not support renaming of the natural key
Option 2: Update the row key for all objects in the stream (headache?)

Elan Hasson

@ElanHasson

@alexgman yevhen/Streamstone#44

Yevhen Bobrov

@yevhen

got custom domain name back http://streamstone.net/ Thanks to the @ElanHasson for donation!!!

Matej Hertis

@mhertis

Special question, what can one do if Stream headers was manually changed in storage and now StreamStone is throwing `Streamstone.ConcurrencyConflictException at Streamstone.Stream+WriteOperation+Batch.HandleStream header has been changed or already exists in a storage :)

Yevhen Bobrov

@yevhen

Reload)

If stream is in memory the only way you can fix the situation is to reload it from storage to sync the etag

Chat via Matrix