These are chat archives for YoEight/eventstore

22nd
Jan 2017
Yorick Laupa
@YoEight
Jan 22 2017 00:33
I fixed #62
because of #61
you were right, those are related in a sense
Yorick Laupa
@YoEight
Jan 22 2017 00:53
finally fixed #61
will push the patch later on sunday
Yorick Laupa
@YoEight
Jan 22 2017 08:18
#62 helped to figure out #61
Akii
@Akii
Jan 22 2017 10:14
nice!
saw your tweet regarding the TMVar
Yorick Laupa
@YoEight
Jan 22 2017 10:57
yeah but it wasn't related to that
bad logic was :-)
took me several hours to figure it out though
laziness wasn't involved
it was a regular race condition
for #62 it was just a logical error. I didn't saved the new checkpoint value in CatchupSMState
hard to test as it requires to lose the connection with the server
Akii
@Akii
Jan 22 2017 11:25
ye
wow
not bad
thanks for fixing both!
I'll probably test that tomorrow
Yorick Laupa
@YoEight
Jan 22 2017 11:25
Thanks for finding those !
please do
Is that you on twitter @AkiiZedd ?
Akii
@Akii
Jan 22 2017 11:29
yes
Akii
@Akii
Jan 22 2017 11:37
thanks! ^.^
Yorick Laupa
@YoEight
Jan 22 2017 11:38
salute is done when salute is due :-)
Akii
@Akii
Jan 22 2017 11:55
I think I just hit my first own evaluation issue
got around 80k events
deriving some state; app takes around 150 mb ram
and then I'm like actually evaluating the thing and it hangs for 25 seconds and ram jumps up to 4.2 gb
fun times
Yorick Laupa
@YoEight
Jan 22 2017 12:10
seems so
Akii
@Akii
Jan 22 2017 12:10
just NFData the whole thing and called force
Yorick Laupa
@YoEight
Jan 22 2017 12:10
particular if you have a lot of record updates
Akii
@Akii
Jan 22 2017 12:10
memory is now constant and it actually computes the value after folding
Yorick Laupa
@YoEight
Jan 22 2017 12:11
I don't have to use NFData 99% of the time though
because I know how GHC strict analysis works
Akii
@Akii
Jan 22 2017 12:11
just playing around atm, trying to get a feel for it
Yorick Laupa
@YoEight
Jan 22 2017 12:12
there are cases where it's needed though
Akii
@Akii
Jan 22 2017 12:30
but I now understand this thunk issue
whatever NFData does by force does not end up consuming 4 GB RAM
so I assume those 4g are thunks
unevaluated thunks
I'm doing a ton of record updates
Yorick Laupa
@YoEight
Jan 22 2017 12:31
that's the issue then
it puts pressure on the GC too
Akii
@Akii
Jan 22 2017 12:38
I don't see any way around record updates
but I definitely see that loading an aggregate with 80k events isn't what you want to do anyway
Yorick Laupa
@YoEight
Jan 22 2017 12:38
there isn't general solution. It depends
Akii
@Akii
Jan 22 2017 12:38
this is just a mockup
heck the eventstore for that project is a 25 MB text file atm xD
Yorick Laupa
@YoEight
Jan 22 2017 13:45
I cut 0.14.0.1 release on Hackage
Akii
@Akii
Jan 22 2017 18:31
actually this is interesting
just say if you think this doesn't belong in the channel
I was thinking that I could "reverse" my stream
Yorick Laupa
@YoEight
Jan 22 2017 18:35
Well considering there isn't much traffic, I don't mind :-)
Akii
@Akii
Jan 22 2017 18:35
let x = [ItemAdded "foo", ItemAdded "bar", ItemAdded "baz"]
data MyProducts = [String]
foldl' getProdName [] x
will yield ["foo", "bar", "baz"]
now make it so that it yields ["baz", "bar", "foo"]
because you use :
and now lets consider take 1 (foldl' getProdName [] x)
this will yield ["baz"]
but what actually happened is that all 3 events were applied
with me so far?
Yorick Laupa
@YoEight
Jan 22 2017 18:38
What this is about ?
Akii
@Akii
Jan 22 2017 18:38
event sourcing
Yorick Laupa
@YoEight
Jan 22 2017 18:38
ah
Akii
@Akii
Jan 22 2017 18:38
let my write that down
I'll just give you a working example
shouldn't be too hard
Yorick Laupa
@YoEight
Jan 22 2017 18:38
what do you try to achieve ?
Akii
@Akii
Jan 22 2017 18:38
lazy loading
without snapshotting
or anything
Yorick Laupa
@YoEight
Jan 22 2017 18:39
alright
so you're suffering from huge memory usage ?
Akii
@Akii
Jan 22 2017 18:39
not suffering at all, just trying to investigate this property
because this very much looks like a mathematical property
"if your aggregate holds x laws you can reverse the stream"
Yorick Laupa
@YoEight
Jan 22 2017 18:40
ok but what lazy loading and snapshotting have anything to do with ?
those are 2 seperates concerns
Akii
@Akii
Jan 22 2017 18:40
let me first come up with an example
Yorick Laupa
@YoEight
Jan 22 2017 18:41
ok go ahead
Akii
@Akii
Jan 22 2017 18:41
I think it's more obvious then
Akii
@Akii
Jan 22 2017 18:46
option1 is your standard way of deriving
so
how does take 1 option1 differ from take 1 option2
so option1 has to go through the entire stream
where as option2 does only need the last event
and with Haskell it will actually "lazy load" the aggregate as you require
Snapshotting tries to optimise this by saving the state as-is every so often
so you can kinda see the relation between lazy loading and snapshotting here
Akii
@Akii
Jan 22 2017 18:52
option1 is what got me 4 GB RAM usage today
(also forget about the performance implications of option2 in this case)
Yorick Laupa
@YoEight
Jan 22 2017 18:57
question is, why do you need to construct a list after consuming your stream ?
Akii
@Akii
Jan 22 2017 18:58
ProductCatalog is just an example state
you're not on DDD EU by any chance
the question really boils down to "what is the latest event you require to build your state"
Yorick Laupa
@YoEight
Jan 22 2017 19:02
what I meant is when I projecting a state over a stream
that projection has never been a list
Akii
@Akii
Jan 22 2017 19:03
forget about the list
Yorick Laupa
@YoEight
Jan 22 2017 19:03
you just want the last element of the stream then ?
Akii
@Akii
Jan 22 2017 19:03
also I don't think this just applies to your regular aggregates
it's more complicated
Yorick Laupa
@YoEight
Jan 22 2017 19:03
fair enough
Akii
@Akii
Jan 22 2017 19:03
in my case I have servers that send 2 events "PlayerJoined" and "MatchChanged"
so what happened today is that I just wanted to show the last 10 matches with their players
so Haskell went back to event 1
and went from there
where as you can actually reverse the stream and build up matches from there
Yorick Laupa
@YoEight
Jan 22 2017 19:04
ok
Akii
@Akii
Jan 22 2017 19:05
I think this is an interesting property
in general
Yorick Laupa
@YoEight
Jan 22 2017 19:05
so if you only care about reading the last 10 events, you only have to read the stream backward
Akii
@Akii
Jan 22 2017 19:05
probably mostly related to statistics
of course
Yorick Laupa
@YoEight
Jan 22 2017 19:05
no need to do it yourself
Akii
@Akii
Jan 22 2017 19:06
but you don't know the amount of events you need
the idea is really to completely rebuild the aggregate from a reversed stream
and just let Haskell lazily fetch events as they're required
Yorick Laupa
@YoEight
Jan 22 2017 19:06
how do you how many events your are interested in ?
Akii
@Akii
Jan 22 2017 19:07
not at all
I let Haskell do that
Yorick Laupa
@YoEight
Jan 22 2017 19:07
nope
Akii
@Akii
Jan 22 2017 19:07
sure
Yorick Laupa
@YoEight
Jan 22 2017 19:07
a lot can be carried out by the server
Akii
@Akii
Jan 22 2017 19:07
The data type looks like this: data Server = Server [Match]
so I basically take 10 matches
Yorick Laupa
@YoEight
Jan 22 2017 19:07
I see
Akii
@Akii
Jan 22 2017 19:08
and Haskell will just continue the foldl until it has 10 or has reached end of stream
Yorick Laupa
@YoEight
Jan 22 2017 19:08
not really
if you need the last 10 matches, you'll need to consume the stream entirely
if it's only 10 random matches
Akii
@Akii
Jan 22 2017 19:09
the last 10 matches are at the end of the stream
Yorick Laupa
@YoEight
Jan 22 2017 19:09
foldl will not end until you reach end of stream
Akii
@Akii
Jan 22 2017 19:09
afaik foldl is the variant that can handle infinite streams
so yes
Yorick Laupa
@YoEight
Jan 22 2017 19:10
if you know where those are, why aren't you reading the stream backward then ?
Akii
@Akii
Jan 22 2017 19:10
this is exactly what I'm getting at
reading streams backwards
Yorick Laupa
@YoEight
Jan 22 2017 19:10
you know there is a function for that in eventstore right ?
Akii
@Akii
Jan 22 2017 19:10
like now that I think about it, this is why getEventStore has that functionality
sure
I'm just exploring the general idea
Yorick Laupa
@YoEight
Jan 22 2017 19:11
ok, you are just brainstorming here :-) ?
Akii
@Akii
Jan 22 2017 19:11
yup
that and get into Haskells laziness
because if Haskell can do that with IO
Yorick Laupa
@YoEight
Jan 22 2017 19:12
be more explicit because I thought you were asking for something more specific
Akii
@Akii
Jan 22 2017 19:12
and I think it can
sorry
^^
Yorick Laupa
@YoEight
Jan 22 2017 19:12
nope, it can't with list at least
you can do it with Data.Sequence
Akii
@Akii
Jan 22 2017 19:12
like the IO op would be "fetch next 500 events"
Yorick Laupa
@YoEight
Jan 22 2017 19:13
yep
Akii
@Akii
Jan 22 2017 19:13
can't Haskell do the first 500 and then do the next
Yorick Laupa
@YoEight
Jan 22 2017 19:13
the driver does it for you
you can specify the batch size you want
and the direction of the read
Akii
@Akii
Jan 22 2017 19:13
ye I know about batch
and that :D
just wondering if you can actually lazy evaluate those IO actions
Yorick Laupa
@YoEight
Jan 22 2017 19:13
I really don't get what you want
which IO actions ?
Akii
@Akii
Jan 22 2017 19:14
fetching those events
Yorick Laupa
@YoEight
Jan 22 2017 19:14
be more precise
Akii
@Akii
Jan 22 2017 19:14
okay
I'm assuming IO [Event]
which is basically defined as readBatch 500 >>= readBatch 500
so you just bind which is (++) for lists and in the end you have all events
but you only take the first 600 events
Yorick Laupa
@YoEight
Jan 22 2017 19:16
hmm
do you ask for something that behaves like Iterator in mainstream programming languages ?
Akii
@Akii
Jan 22 2017 19:17
and now my question: does Haskell go to the end of the stream performing all IO actions or does it stop after 2
ye that'd probably a good analogy
Yorick Laupa
@YoEight
Jan 22 2017 19:18
well I wrote something like this in a higher level library of mine (available in Hackage)
Akii
@Akii
Jan 22 2017 19:19
I said IO [Event] but it's actually [IO [Event]]
you can sequence that to get IO [Event]
correction: that's almost sequence
once you get StreamIterator
you can use iteratorNext
which does the lazy loading for you
Akii
@Akii
Jan 22 2017 19:21
interesting :D
Yorick Laupa
@YoEight
Jan 22 2017 19:22
in this case, you will never have more that 500 events in memory
there's an implementation for my GetEventStore driver
The thing is I didn't implement backward read with this interface
Akii
@Akii
Jan 22 2017 19:24
well it's not a common use-case
not so common at least
Yorick Laupa
@YoEight
Jan 22 2017 19:37
writing it is trivial though