Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 31 2019 19:32

    prolic on v1.10.3

    (compare)

  • Jan 31 2019 19:32

    prolic on master

    update changelog (compare)

  • Jan 31 2019 19:29
    prolic closed #176
  • Jan 31 2019 19:28
    prolic commented #186
  • Jan 31 2019 19:28
    prolic closed #186
  • Jan 31 2019 19:28

    prolic on master

    fix restarting projection durin… respect lock in memory resetting projection and 12 more (compare)

  • Jan 31 2019 16:26
    fritz-gerneth commented #189
  • Jan 31 2019 16:21
    prolic commented #189
  • Jan 31 2019 16:06
    sandrokeil commented #189
  • Jan 31 2019 15:31
    grzegorzstachniukalm synchronize #186
  • Jan 31 2019 15:23
    kamil-nawrotkiewicz edited #76
  • Jan 31 2019 15:22
    kamil-nawrotkiewicz edited #76
  • Jan 31 2019 15:22
    kamil-nawrotkiewicz opened #76
  • Jan 31 2019 15:19
    grzegorzstachniukalm synchronize #186
  • Jan 31 2019 15:09
    grzegorzstachniukalm synchronize #186
  • Jan 31 2019 13:54
    coveralls commented #186
  • Jan 31 2019 13:48
    mrook commented #189
  • Jan 31 2019 13:41
    prolic commented #189
  • Jan 31 2019 13:40
    prolic commented #189
  • Jan 31 2019 13:31
    fritz-gerneth commented #189
Fritz Gerneth
@fritz-gerneth
@bweston92 we're using the pdo-event-store / projections. We don't replay events, but start consuming streams from the beginning
Bradley Weston
@bweston92
Ahh ok
Fritz Gerneth
@fritz-gerneth
( delete / recreate essentially )
Bradley Weston
@bweston92
So here is what I have currently, but with way more contexts, each cloud is a separate project basically and each rectangle is a deployment
download.png
Bradley Weston
@bweston92
however as I am now getting more and more contexts I think the eventstore will probably restrict me a little bit
and think the following might be better
download (1).png
Fritz Gerneth
@fritz-gerneth
i.e. add a publisher in between to off-load work from the ES to a queue
I've been playing with this a bit myself, mostly because mysql does not support subscriptions
the two major things to take care of here are:
  • how can you detect gaps in your stream (i.e. ensure messages are consumed in order, not every queue guarantees this)
  • how can you get access to historic events (i.e. when you create a new projection and want to / need to start from the very beginning of a stream)
so I was/am aiming for a hybrid approach, use event store to get up to date, then stay up to date with pubsub
Bradley Weston
@bweston92
so them 2 issues you've explained, I have thought about both with no real solution other then write tooling
Fritz Gerneth
@fritz-gerneth
nothing there out of the box, yes
Bradley Weston
@bweston92
event order can be determined by aggregate identifiers and versions and if there is a missed event send it away for a while until it gets it, if not then report an error I suppose? but then when replaying would use a lot of resources to handle this
and the second one would be to send directly to a subscription from all the stream sources it cares about
Fritz Gerneth
@fritz-gerneth
tbh.. I gave up on this after a while.. I had ideas how to handle all these, but the effort wasn't just worth it
Bradley Weston
@bweston92
is the trade off worth it :'(
just about to say that yeah
but the more I get eventstore to do the slower and harder it will get
Fritz Gerneth
@fritz-gerneth
do you think this or measured this?
Bradley Weston
@bweston92
bit of both, eventstore when having lots of running projections aggregating different streams becomes slow
Fritz Gerneth
@fritz-gerneth
we have about 100 projections running and we still can scale the ES (aws rds instance)
but then I don't know the eventstore.org :)
you can still start splitting them up, but have read models on ES directly
from the graph you're doing both at the same time?
Bradley Weston
@bweston92
ES has the projections that aggregate streams, the resulting streams from that aggregation a projection that writes it to a local read model uses a persistent subscription on the result stream
Fritz Gerneth
@fritz-gerneth
we did delegate this in a query-bus instead
Bradley Weston
@bweston92
the query bus would only read from the local projection
Fritz Gerneth
@fritz-gerneth
do they need to?
Bradley Weston
@bweston92
yes, because that is where the data is formatted for each api endpoint response type
Fritz Gerneth
@fritz-gerneth
formatted = stored I assume?
Bradley Weston
@bweston92
stored in the way the consumers of that endpoint will want it
@prolic / @codeliner you have any thoughts on this?
Fritz Gerneth
@fritz-gerneth
all in all it sounds like an issue scaling the eventstore to more projections though
Bradley Weston
@bweston92
yes, but also means I don't have to use event sourcing for everything either :| for smaller stuff I could just just an outbox that pushes it onto a message broker
Fritz Gerneth
@fritz-gerneth
that's the part where we use the MQ, for anything that doesn't need to be replayed in the future :)
Bradley Weston
@bweston92
but what if some of your projections consume both an event stream that can be replayed and a standard MQ?
Fritz Gerneth
@fritz-gerneth
for us, they don't, one is a projection updating a read model, the other a standaline event listener
Bradley Weston
@bweston92
for example a customer is event sourced but how many times they've clicked a link isn't. The projection is total clicks by a customer. Customer created event (from event sourced) inserts with 0 clicks then clicks from the MQ increment that
Fritz Gerneth
@fritz-gerneth
that'll introduce some serious concurrency pain :)
Bradley Weston
@bweston92
so what would the solution be?
store the events locally?
Fritz Gerneth
@fritz-gerneth
use only one of it I guess
Bradley Weston
@bweston92
the events from the MQ store them in a local event store?
just imagining all the backup and restoration to put in place and test
Fritz Gerneth
@fritz-gerneth
that sounds like you're trying to re-invent ES-replication then (read-instances)
Bradley Weston
@bweston92
doing what I suggested would end up doing that yeah, just trying to think of a way round it all
Fritz Gerneth
@fritz-gerneth
I'd probably try to solve this issue on the ES side, with read-replicas on their side
or split up into different ES instances