These are chat archives for YoEight/eventstore

18th
Jan 2017
Akii
@Akii
Jan 18 2017 10:30
pretty sure I'm doing things wrong @YoEight
Akii
@Akii
Jan 18 2017 10:52
I'm really having trouble dealing with all the exceptions lol
Akii
@Akii
Jan 18 2017 11:01
try $ shutdown es >> waitTillClosed es :: IO (Either SomeException ())
this seems to be a reliable way of shutting down the connection in any case :D
if it ever comes to that I pretty sure don't care about the reason anymore
Akii
@Akii
Jan 18 2017 11:30
this is basically what I have currently for subscriptions: http://lpaste.net/351390
Akii
@Akii
Jan 18 2017 11:42
now the main idea is to always come to an end. If the app is closed normally, I will unsubscribe the subscriptions
if anything happens during the IO action, the subscription must also be closed and the error logged; last processed event saved
stuff like that :D
Yorick Laupa
@YoEight
Jan 18 2017 13:30
I try to understand what you try to do but here's some important notes
the only time you should call shutdown is when your application is about to terminate
GetEventStore connection is meant to remain open during the entire life of your application
On the same token, there is no need to call unsubscribe in your case. If an exception has been raised by the subscription, it's already been unsubscribed
Yorick Laupa
@YoEight
Jan 18 2017 13:35
There is no bookkeeping needed on the user side with GetEventStore connection
GetEventStore /= SQL Connection
there is nothing to clean on your side
everything is handled by the driver internally
About your code snippet
this isn't the most general way to fold an subscription
Yorick Laupa
@YoEight
Jan 18 2017 13:41
quite the opposite, because you can only perform one kind of effect, here IO
your also very tight to one kind of subscription state
here's a more general
Akii
@Akii
Jan 18 2017 13:42
hi :D
dang I gotta run in 3 mins
Yorick Laupa
@YoEight
Jan 18 2017 13:43
runSubscription :: MonadIO m => Subscription Catchup -> (RecordedEvent -> m ()) -> m ()
Akii
@Akii
Jan 18 2017 13:43
regarding unsubscribe: I'm not entirely sure which exception would stop the subscription
(ok, that's indeed better)
so there are these exceptions like "connection gone (max retry attempts)" those definitely stop the subscriptions
Yorick Laupa
@YoEight
Jan 18 2017 13:44
because those exception you catch come from the driver itself
Akii
@Akii
Jan 18 2017 13:44
and then there are exceptions thrown in my RecordedEvent -> m() function
exactly
Yorick Laupa
@YoEight
Jan 18 2017 13:44
either because the server said something went wrong
or because it can't reach out with the server anymore (depending on the retry strategy)
Akii
@Akii
Jan 18 2017 13:45
I think I'll let the driver try this forever
keepRetrying
seems like a sane default
anyway, got to run now; I'll be back asap
Yorick Laupa
@YoEight
Jan 18 2017 13:47
If an exception is thrown by your callback, that's not the driver business, but seems ludicrous for me to say, "I want to unsubscribe in this case". that looks like a business logic bug
Akii
@Akii
Jan 18 2017 13:48
well my thinking here is that something went wrong
especially when handling events the possibility is high that I fucked up
with the implementation
and I'd rather have it stop and send me a mail "fix this"
of course that's not the drivers problem :D
Yorick Laupa
@YoEight
Jan 18 2017 13:49
exactly, so in this case, you should let it crashes
Akii
@Akii
Jan 18 2017 13:49
I just need to stop then
Yorick Laupa
@YoEight
Jan 18 2017 13:49
for you can work on it later :-)
Akii
@Akii
Jan 18 2017 13:50
well I don't want to let everything just crash
commands can still be handled
other streams can still be handled
Yorick Laupa
@YoEight
Jan 18 2017 13:50
how do you know you had a serious bug if you keep reviving the same code ?
Akii
@Akii
Jan 18 2017 13:50
not reviving
just stopping this particular subscription
like I'm integrating with other bounded contexts here
this is not a complete failure
got to run now though sry
Akii
@Akii
Jan 18 2017 14:18
aaand I'm back ^^
so what I can take from this is basically: no need to do anything manually with the driver
Akii
@Akii
Jan 18 2017 14:30
MonadIO does not allow catching exceptions unfortunately
Yorick Laupa
@YoEight
Jan 18 2017 14:30
You don't get it
You can use any monad you want as long as it as a MonadIO instance
there are a lot of those that can catch :-)
catching is orthogonal to iterate over your subscription
let the specialized monad do it for you
Akii
@Akii
Jan 18 2017 14:36
well I can catch from outside the function then
but catches is bound to IO
Yorick Laupa
@YoEight
Jan 18 2017 14:36
nope
you can do it your callback too
you can also use IO you know, because IO has a MonadIO instance
Akii
@Akii
Jan 18 2017 14:37
yes but then I would not be able to stop processing the events
Yorick Laupa
@YoEight
Jan 18 2017 14:37
no, you can still stop processing the events
if you throw an exception in your callback, it will not proceed
that's part of if Monad implementation
Akii
@Akii
Jan 18 2017 14:39
I see
sorry if I sound like the biggest noob
Yorick Laupa
@YoEight
Jan 18 2017 14:39
Being a noob is a transient state, not a persistent one
We all went through this one time :-)
Akii
@Akii
Jan 18 2017 14:40
ye xD
I just want to be as little annoying as I can be
need to read up on this
Yorick Laupa
@YoEight
Jan 18 2017 14:41
That's thoughtful :-)
don't worry
Yorick Laupa
@YoEight
Jan 18 2017 14:41
Ask question, I'll do my best to answer you
Akii
@Akii
Jan 18 2017 14:42
thanks :D
I've some basic intuition about all of this
but I just can't see how I would end processing the subscription in a meaningful way
Yorick Laupa
@YoEight
Jan 18 2017 14:42
Master it, after that a lot of Haskell code will make sense to you
Akii
@Akii
Jan 18 2017 14:43
it's also very important that I get a hold of the last processed event
would you do that inside the callback?
Yorick Laupa
@YoEight
Jan 18 2017 14:43
StateT could help here
Akii
@Akii
Jan 18 2017 14:43
sure state, but for now I'm just passing it around manually
Yorick Laupa
@YoEight
Jan 18 2017 14:43
and it also has a MonadIO instance too
Akii
@Akii
Jan 18 2017 14:44
okay, I will now play around with that
Yorick Laupa
@YoEight
Jan 18 2017 14:44
that's how you do polymorphic programming in Haskell
Akii
@Akii
Jan 18 2017 14:44
yep
like I know :D
just don't see it working in this case yet
Yorick Laupa
@YoEight
Jan 18 2017 14:44
you will
Akii
@Akii
Jan 18 2017 14:44
yep
I'll just fix that to MonadIO now and see what I can come up with
Akii @Akii is certain the "aha!" effect will come shortly
Akii
@Akii
Jan 18 2017 15:24
I now have
runSubscription :: MonadIO m => Subscription Catchup -> (RecordedEvent -> m a) -> m a
runSubscription sub f = forever $ (liftIO $ resolvedEventOriginal <$> nextEvent sub) >>= f
and this callback
printState :: MonadIO m => RecordedEvent -> StateT Int m ()
printState _ = do
  modify (+1)
  thingy <- get

  liftIO $ if thingy == 2
    then throwIO Foo
    else print thingy
and then
a <- async $ flip execStateT 0 $ runSubscription sub printState
handle (\(e :: Foo) -> putStrLn "now what" >> return 0) $ waitAsync a
which is almost good
(I can re-arrange the waitAsync to happen inside the async, so no need to "wait for an exception" - just convenient here)
the only thing I can not get a hold off is the last processed event
could stick it into a TVar
but then I don't really need State
(again my goal is to save the last processed event for a given subscription should it end; doesn't matter how)
Yorick Laupa
@YoEight
Jan 18 2017 15:28
in this case I recommend an IORef
Akii
@Akii
Jan 18 2017 15:29
I've read that term before
Yorick Laupa
@YoEight
Jan 18 2017 15:29
TVar are too overkill (and aren't cheap) for that
Akii
@Akii
Jan 18 2017 15:29
interesting
but I definitely see how this is much better now
Yorick Laupa
@YoEight
Jan 18 2017 15:30
good
Akii
@Akii
Jan 18 2017 15:30
I can almost see the composability
like deduplication; State
deserialization of the event; another function
Yorick Laupa
@YoEight
Jan 18 2017 15:31
a lot can be done
Akii
@Akii
Jan 18 2017 15:31
and then just compose that stuff
thanks, again ^.^
Yorick Laupa
@YoEight
Jan 18 2017 15:32
:-)
Akii
@Akii
Jan 18 2017 15:32
btw, one thing I noticed
when you have a running subscription and set the option to retry connecting forever
then kill the server and restart it
there is a chance the server will re-send events
Yorick Laupa
@YoEight
Jan 18 2017 15:33
nope
Akii
@Akii
Jan 18 2017 15:33
yes ^^
had that multiple times, resends around 2
Yorick Laupa
@YoEight
Jan 18 2017 15:33
nope
that's my driver
Akii
@Akii
Jan 18 2017 15:33
ah ok
Yorick Laupa
@YoEight
Jan 18 2017 15:33
I keep track of where you were at
Akii
@Akii
Jan 18 2017 15:34
nifty!
Yorick Laupa
@YoEight
Jan 18 2017 15:34
I asked to restart from that point
thanks
Akii
@Akii
Jan 18 2017 15:34
but those events were already processed
(I was just printing them out so I know)
Yorick Laupa
@YoEight
Jan 18 2017 15:34
I does it only for Catchup subscription
Akii
@Akii
Jan 18 2017 15:35
do you simply "restart" the subscription at the initial given number or do you keep track as you go?
Yorick Laupa
@YoEight
Jan 18 2017 15:35
if you want a guarantee to process only one time an event, you need Persistent subscription
I keep track as I go
Akii
@Akii
Jan 18 2017 15:35
btw, not a problem since I can just keep track of the msgs already processed
and I'm aware about the persistent subscriptions
Yorick Laupa
@YoEight
Jan 18 2017 15:36
ok cool
Akii
@Akii
Jan 18 2017 15:36
persistent subs are at-least once
so even then no guarantee
Yorick Laupa
@YoEight
Jan 18 2017 15:36
what ?
what do you mean ?
Akii
@Akii
Jan 18 2017 15:36
guarantee to process only one time an event
sounded like exactly once
Yorick Laupa
@YoEight
Jan 18 2017 15:37
a.k.a you have a guarantee then
The server takes care of this. By keeping track of the state on its side
Akii
@Akii
Jan 18 2017 15:38
yep
Yorick Laupa
@YoEight
Jan 18 2017 15:38
it handles timeout too
Akii
@Akii
Jan 18 2017 15:38
but on message level the guarantee is at least once
so it can be that you get one event twice
Yorick Laupa
@YoEight
Jan 18 2017 15:38
you can't have it twice
Akii
@Akii
Jan 18 2017 15:38
fx. if you don't acknowledge it
once it's processed it won't come again
we're probably talking about different things
doesn't matter anyway :D
Yorick Laupa
@YoEight
Jan 18 2017 15:39
if you don't acknowledge it, you can have it more that one then
Akii
@Akii
Jan 18 2017 15:39
yep
Yorick Laupa
@YoEight
Jan 18 2017 15:39
possible :-)
Akii
@Akii
Jan 18 2017 15:39
anyway, really good
Yorick Laupa
@YoEight
Jan 18 2017 15:39
what do you mean ?
Akii
@Akii
Jan 18 2017 15:39
I like how I can just sit there and forever try to get the connection back
works great at least for subscriptions
will have to investigate what to do with sending events
Yorick Laupa
@YoEight
Jan 18 2017 15:40
I can tell you :-)
Akii
@Akii
Jan 18 2017 15:40
does it just abort the async after a while?
Yorick Laupa
@YoEight
Jan 18 2017 15:41
depends
when you sendEvents, that request is only queued
so if the connection dropped before it get sent, the driver will retry
if the command has been sent and the connection dropped, on waitAsync you'll have an exception
if the connection hasn't dropped but take too much time, we'll stilll wait
until the end of the universe
Akii
@Akii
Jan 18 2017 15:44
interesting
Yorick Laupa
@YoEight
Jan 18 2017 15:44
what I want to do is to track if an operation takes too much time
like the C# driver
It's on my plate but haven't start any dev on it yet
Akii
@Akii
Jan 18 2017 15:45
well if I get an exception during event sending I'll just log them to a file
ha now I'm curious
I wonder if I can write something that will just keep failed events in memory and then tries to send them again once the store is back
so like you're sending, store dies (or whatever), instead of just dying I queue events
and then the store comes back up
Yorick Laupa
@YoEight
Jan 18 2017 15:52
I don't expect you to queue anything, that's what the driver does on your behalf
Akii
@Akii
Jan 18 2017 15:53
sure, but the driver can only handle so much
so it will throw an exception if the connection is gone
but I can't just let my system die then
in fact I can proceed to happily work without the store
Yorick Laupa
@YoEight
Jan 18 2017 15:53
you can't know if the connection is gone
Akii
@Akii
Jan 18 2017 15:53
subscriptions will be waiting until the connection is back
meanwhile the driver will retry
but I need to still be able to "send" events
do you know what I mean?
Yorick Laupa
@YoEight
Jan 18 2017 15:54
you can't know for a fact if it waits because there is no more events coming or because we try to reconnect to the server
All you have to know
don't worry about the connection
send whenever, wherever you want
if your command failed, you'll be notified
Akii
@Akii
Jan 18 2017 15:56
with the exception in the async
Yorick Laupa
@YoEight
Jan 18 2017 15:56
async is on the user side
an exception in the user side doesn't mean the driver is no longer working
an execption with asyncWait is what I call being notified
Akii
@Akii
Jan 18 2017 15:58
well that just waits for the async
that's what I meant with "the exception in the async"
Yorick Laupa
@YoEight
Jan 18 2017 15:59
ah sorry, I thought you said except
instead of exception
Akii
@Akii
Jan 18 2017 15:59
I'm going to try this now :D
always have to make sure I understand what's going on
Yorick Laupa
@YoEight
Jan 18 2017 16:01
You could also have a look at the code
the Truth is there :-D
Akii
@Akii
Jan 18 2017 16:07
I've the feeling that the driver will wait forever :D
s_retry = keepRetrying
Akii
@Akii
Jan 18 2017 16:15
okay I'm beginning to get a better understanding of what's happening
if I use default settings the connection dies (as expected)
but then I also need to restart everything related to the driver
makes sense
so ok, this is then probably where an intelligent retry functionality would make sense
interesting
Akii
@Akii
Jan 18 2017 16:20
now I see what you meant with "retrying a failed subscription if there is a bug"
so there are definitely cases where I can resume a previously failed subscription (e. g. if the connection died)
but if I had a logical error I cannot just restart the subscription
same with when my database connection drops (I'm not event sourcing so GetEventStore isn't the only DB)
but in that case I might as well just let the app crash
I can live without the event store but not without the aggregate store
the puzzle pieces begin to fit together
Yorick Laupa
@YoEight
Jan 18 2017 16:38
Cool
however, a logical error is also called a bug
and again if you have a logical error as you said, you don't need to restart anything because the subscription hasn't died
with keepRetrying, it will retry until the end of the universe
so I don't see why you want to write code to handle the use case of a connection dropped
because one you can't know and second, that's irrelevant to your application
just use the right settings in the driver setttings and happy hacking !