These are chat archives for RBMHTechnology/eventuate

1st
Dec 2016
Martin Krasser
@krasserm
Dec 01 2016 06:56 UTC
Hi @benash, you can share a single Cassandra cluster across shards but only if each shard has its own event log in the Cassandra cluster. Without further replication, this will give you event collaboration between EventsourcedActors within a shard. If you additionally want to have cross-shard event collaboration you need to connect the shard-specific local event logs to a replicated event log. If you don't need event collaboration between EventsourcedActors but only want to aggregate events from all shards into a single log (for creating views, for example) you could connect the shard logs to an aggregate logs and run EventsourcedViews and/or EventsourcedWriters there. Hope that helps.
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:09 UTC

Hi! i am fairly new to functional programming, event sourcing or even scala programming (since i had been working mostly on Android development in Java over the past years). But i have been studying a lot on the past months in order to select a technology stack for rewriting a backend for an app and i have decided to choose event sourcing with eventuate as the persistence layer for the project :).

I think i have read most (if not all) the documentation i have found on eventuate but i still have several doubts about the proper/best way to do some things and maybe you could give me some advice. The part i am stuck now is how to assign ids to entities or aggregates and how to assign them to actors.

The example application, for instance, uses manually assigned ids for the orders and it can easily forward each order to an actor based on that id (since it is known from the very beginning). But in my case the data is coming from external systems and in the current backend the entity id is retrieved from several possible queries to the database based on the parsed data from the external systems (the parsed data is also often incomplete which is the reason for the alternate queries). If a row is found by these queries, the new data is considered to be for the same entity (with its corresponding id). If none of the queries have a result, a new entity (with an auto incremented id) is added to the database.

The example application on eventuate also has a querydb package that creates customer ids by incrementing a counter, although the tutorial seems to be missing an explanation on that part of the eventuate-examples. Despite that, that approach seems to be a good alternative for the auto incremented ids of the relational database on the current backend although i probably need to include on it the queries for previous entities and not just the incremental counter. Therefore, my question is how to do that.

So far, my best idea is to have the full state for all the entities in this actor (an actor equivalent to the Emitter actor of the querydb example) but i am not sure if that is a good practice. Another alternative may be to query another view from this actor and then forward the events to additional actors for each entity once i know their ids.

Finally, the ids i have now are integer/long ids, but the actors on eventuate require unique string ids. Is there a recommended policy for assigning entity ids to actor ids? There are also more than one entity types on the same backend that currently have integer ids too, so a simple conversion of the ids to string may not be good enough.

What do you think? Any tips?

Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:13 UTC
hi @jrgonzalezg , I'm also relatively new to Eventuate and currently working on a toy project which uses Eventuate as an event store
in the past I have mostly worked with Java or Java EE based applications and the typical enterprise setup with relational databases (or worse :-)) for persistence
comparing that experiences with the usual approaches like IDs generated by the database itself I can highly recommend to use client-generated IDs if possible
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:16 UTC
hi! :D, so we both are looking for better ways to do things and both ended up here :D. The client IDs are not possible, because all the data is from that external systems and i have no control at all over them..
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:17 UTC
if you are already using these IDs as your internal IDs I think they already are somewhat like client-generated IDs
maybe it's not optimal but you could consider the external system as one client of your application which already provides IDs for entities
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:19 UTC
But that is the issue, their data does not have IDs, just inconsistent, partially incomplete related feeds with some metadata plus some data that changes over time
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:19 UTC
but your IDs are derived from that information?
btw. IDs that change over time sounds very strange :-)
I think you have to find a way to get stable IDs for your enities inside your application
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:21 UTC
not exactly, i try to assign an id based on that information and when i receive new data i try to incorporate them to the entities i created before if they exist
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:21 UTC
then you can use these IDs as aggregateIds inside Eventuate's EventSourcedActors and everything will be find
fine
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:22 UTC
the IDs should not change if i can properly identify the data as being for the same entity :D, sometimes i can't but that is not the main problem
to simplify it, it is like if they were places
sometimes i get an address, sometimes a lat,lng, sometimes both
but i do not get ids ever from them
i do queries based on what i get (like an address) and then i have to identify if i have seen that place before from the queries i make
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:24 UTC
well, I'm not an expert but may Eventuate or using an event store is not the tool you need for your application
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:24 UTC
the thing is that besides the id part, it matches very well the event sourcing ideas
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:24 UTC
the idea behind Eventuate's EventsourcedActors is to model something very similar to an aggregate in domain-driven design
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:25 UTC
because i have thinks like daily data for each place
that can be modelled as events
these places are the aggregates in the domain driven design sense
and there are a lot of events for each of them
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:25 UTC
ok, but then it seems that "place" is a better candidate for an Entity or Aggregate
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:26 UTC
yes, to simplify it, you can consider that the place is the aggregate in this case
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:26 UTC
the basic way event sourcing and Eventuate works is that you identify Entities and Aggregates which definitely should have a unique identity
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:27 UTC
(it is actually composed of several tables / entities)
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:27 UTC
it's not problem to have some kind of nested data structure to represent an Aggregate
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:27 UTC
yes, i know
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:27 UTC
but it should have an identity
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:28 UTC
the unique identity is the id i assign to the places
but it is only on my system
not on the external systems where the data comes from
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:28 UTC
so that every event for that "place" can be assigend to the corrent event stream for the aggregate in question
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:28 UTC
they do not give proper ids
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:28 UTC
i think that's fine
relying on IDs from external systems is not a good idea anyway
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:29 UTC
but i can not compute the internal ID from the data i receive, there is no consistent way to do it
that is why i need the queries
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:29 UTC
and you also should model changes to your places as commands and domain events
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:29 UTC
yes, i already have that part
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:29 UTC
I think otherwise event sourcing or CQRS doesn't make much sense
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:30 UTC
i already have commands and events that make sense
but with a hole to fix :D (the id)
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:31 UTC
then it shouldn't be a big deal even if it's a complicated process to generate your IDs (by issuing some queries or something like that)
you just have to identify the incoming request in some way
an of course it should be possible to reproduce the same ID for the same place for later requests
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:32 UTC
that is what i tried to describe on my first message
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:32 UTC
because stored events in Eventuate are "grouped" by this aggregate ID
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:33 UTC
the only way i have to reproduce the ID is by querying against all the current places
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:33 UTC
ok, so that's a different problem not directly related to Eventuate or event sourcing
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:33 UTC
well, it is related to the migration to eventuate / event sourcing
the way to do it on the relational system is trivial
my question is about a recommendation on how to do it on eventuate / event sourcing
or if there is something better than the initial idea i have described
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:35 UTC
well, my application is structured like the typical onion architecture
and I can just generated UUIDs in the application layer for each request which creates a NEW aggregate instance
afterwards each client request has to supply the same UUID
and I'm using the toString() representation as the aggregateId inside EventsourcedActors
so it's just the ID of my aggregates
and I think you could basically do something similar
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:36 UTC
you just have one type of aggregates?
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:37 UTC
no, I have multiple types of aggregates
for each aggregate type there is a specific implementation of an EventsourcedActor
and each INSTANCE of an EventsourcedActor (no matter which type) must have a unique ID
which is similar to an entity ID in a relation data model with an O/R mapper
events are then stored for each ID
so each aggregate has its own event stream
which of course makes sense
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:39 UTC
yes, but if you have things like Aggregate1(id: Int) and Aggregate2(id: Int) you can not use id.toString for the eventSourcedActor
because the will collide for the two aggregates
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:39 UTC
well, of course you have to use something like id.toString()
the ID is a kind of key
it has to be unique for each aggregate instance
for you that probably means each "place" should have its own internal unique ID
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:40 UTC
i currently have unique keys for instances of each aggregate but aggregate1 and aggregate2 can have instances with the same id
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:41 UTC
are you familiar with the concepts of DDD? maybe we are using the same name for different concepts :-)
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:41 UTC
i do not query aggregate1 with ids of aggregate2 or the contrary
maybe i am wrong but i have not seen that DDD requires to have unique ids across aggregates but on each one separately
Place with unique id 1 and Product with unique id 1 for instance
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:42 UTC
yes, that's what I'm talking about
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:43 UTC
but you can not create actor 1 to handle place 1 and product 1
you may have to do actor named "place" + id
or "product" + id
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:43 UTC
I'm not 100% sure at this point
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:43 UTC
my question for that part is about that
if it is recommended to use a prefix
or some other approach
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:44 UTC
in practice you would most probably have different implementations of an EventsourcedActor for Product and for Place
and I'm not sure if Eventuate requires aggregateIds to be unique across all types of actors
or just across all instances of the same type of actor
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:44 UTC
i think so
but i have read so many things that maybe that requirement was for other id and not the actors :D
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:45 UTC
that's where my knowledge probably is not enough :-)
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:46 UTC
well, thanks anyway!, let's see if somebody else can help :)
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:46 UTC
I haven't had this problem so I'm really not sure if Eventuate also considers the type of an actor when it stores events
but I could imagine that the type of an actor is also stored alongside the event data
which would solve the problem with confliciting IDs across different aggregate types
global unique id
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:48 UTC
ok, makes sense, but then you have a problem with this approach
or you have to use something like the prefix you mentioned
or you could also use something like an UUID
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:49 UTC
yes, as i said i just want some opinions or to know how people are doing similar things
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:49 UTC
I guess it doesn't have to be an int
of course @krasserm is the expert
I haven't use Akka persistence or Eventuate before the application I'm currently building
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:51 UTC
it is my first one too, and probably i did not started with an easy one to model :D
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:51 UTC
but it looked really promising right from the start and I'm not disapointed that I have chosen to give it a try ;-)
I think event sourcing and CQRS is a very interesting approach for a lot of applications
and it's surely not more complicated than using an O/R mapper correctly
you just have to change the way you design and think about your applications
Juan Ramón González
@jrgonzalezg
Dec 01 2016 20:52 UTC
i agree with that :)
Marco Ehrentreich
@mehrentreich
Dec 01 2016 20:56 UTC
and you shouldn't forget that it may be more work to implement EventsourcedViews than it is to implement EventsourcedActors to store events
depending on the kind and number of queries you need to make
on the other hand it's really nice to have cleanly separated model for the write and read side of your application
Juan Ramón González
@jrgonzalezg
Dec 01 2016 21:00 UTC
yes, i will continue thinking about it...
Marco Ehrentreich
@mehrentreich
Dec 01 2016 21:02 UTC
I would recommend to create a simple prototype application to see all the parts in action
that helped me to quickly decide if Eventuate or event sourcing can really solve my problem
Juan Ramón González
@jrgonzalezg
Dec 01 2016 21:05 UTC
well, the two approaches i described i think that could work, but i am thinking about better alternatives before deciding between them
Marco Ehrentreich
@mehrentreich
Dec 01 2016 21:07 UTC
from what I understood so far your core problem sounds more like a classification problem
to match an incoming stream of location data to specific places
so maybe it's more a stream processing problem
Juan Ramón González
@jrgonzalezg
Dec 01 2016 21:08 UTC
it is actually both, and there are more things after that, but i think that they are not relevant for this part
to me event sourcing and streaming also have similarities and i see them as complimentary
Marco Ehrentreich
@mehrentreich
Dec 01 2016 21:10 UTC
yes they surely are
I guess that's why there's also a Spark adapter which is part of Eventuate to further process events
Juan Ramón González
@jrgonzalezg
Dec 01 2016 21:12 UTC
I am actually trying to mix DDD+FP+Event Sourcing+Reactive Systems+Microservices+...
Marco Ehrentreich
@mehrentreich
Dec 01 2016 21:13 UTC
that sounds like an interesting but very ambitious goal :-)
Juan Ramón González
@jrgonzalezg
Dec 01 2016 21:14 UTC
that's why i have had to study so much and still more to go :D
Marco Ehrentreich
@mehrentreich
Dec 01 2016 21:14 UTC
that sound's familiar to me
Juan Ramón González
@jrgonzalezg
Dec 01 2016 21:14 UTC
:)
Marco Ehrentreich
@mehrentreich
Dec 01 2016 21:14 UTC
it's easy to forget to get started with something
Juan Ramón González
@jrgonzalezg
Dec 01 2016 21:16 UTC
nice talk BTW, i have to go to dinner now (22PM here) and later i will come back to see if there are more comments :)
Marco Ehrentreich
@mehrentreich
Dec 01 2016 21:16 UTC
the community here in this channel is actually quite quiet
but hopefully there are at least a few other opinions
I'd be glad to hear from your experiences