These are chat archives for akkadotnet/akka.net

28th
Sep 2017
Thomas Denoréaz
@ThmX
Sep 28 2017 09:50
Hi! I'm working with Akka.IO in order to send byte array through a TCP connection, however even though the socket buffer size is higher (50K), the Tcp.Received only receives 256 bytes (or less) and my problem is that I'm sending data bigger than 1M meaning that the Socket is overflooded and the socket is reset
I tried sending only few kilobytes instead of megabytes and then it is working. I was hoping that Akka.IO would automatically handle the buffering :-/ Any ideas?
Vagif Abilov
@object
Sep 28 2017 12:32
@kariem-ali we also started getting this error often after upgrading to Akka 1.3.1. And we are using different database (SQL Server). Same exception:
[Error] Persistence failure when replaying events for persistenceId ["..."]. Last known sequence number [0]
Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: ?. Path '', line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue()
It's also coming from SnapshotStore
kariem-ali
@kariem-ali
Sep 28 2017 12:47
@object Yes, it's not unique to Sqlite. It's in Sql.CommonAkka.Persistence.Sql.Common.Snapshot.QueryExecutor.GetSnapshot()which gets the serializer through Serialization.FindSerializerForType(type, Configuration.DefaultSerializer); which returns the Akka.Serialization.NewtonSoftJsonSerializer (the wrong serializer). I am digging into it myself and will report the issue if I can pinpoint it. Have you done any digging into this yourself?
From what I can tell so far. It disregards the serializers specified in the configuration akka.actor.serializers
Vagif Abilov
@object
Sep 28 2017 13:06
@kariem-ali I didn't dig further into this issue. Here's our serialization section:
serializers {
hyperion = "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
json = "Akka.Serialization.NewtonSoftJsonSerializer"
}
serialization-bindings {
"System.Object" = hyperion
"Newtonsoft.Json.Linq.JObject, Newtonsoft.Json" = json
}
Jose Carlos Marquez
@oeaoaueaa
Sep 28 2017 13:09
Hi, just submitted a pull request for Akka.DI.Ninject to upgrade it to 1.3.1 AkkaNetContrib/Akka.DI.Ninject#10
Vagif Abilov
@object
Sep 28 2017 13:09
If this is as you are saying, we should be able to reproduce it
Paweł Bańka
@pmbanka
Sep 28 2017 13:19
@kariem-ali @object this looks fishy https://github.com/akkadotnet/akka.net/blob/a478c5e8c8d5de97dcf7bcea7e2a80fbaa5e6cc8/src/core/Akka/Serialization/Serialization.cs#L263 because if I'm not mistaken, it looks for a serializer... but filters out serializer for Object type (Hyperion in our case).
but before that, the serializer is updated from GetSerializerByName(), so it is not null anymore
so I guess this is the issue
Paweł Bańka
@pmbanka
Sep 28 2017 13:25
My guess is that here https://github.com/akkadotnet/akka.net/blob/cc73cdc87949849a68ffc1a6a0376033d473c46e/src/contrib/persistence/Akka.Persistence.Sql.Common/Snapshot/QueryExecutor.cs#L330 instead of passing Configuration.DefaultSerializer, null should be passed as a second argument
(edited messages above so that it makes more sense)
Vagif Abilov
@object
Sep 28 2017 13:29
@pmbanka but was this recently changed? Browsing history, can't find what commit could have broken it.
Paweł Bańka
@pmbanka
Sep 28 2017 13:29
didn't check the history
checking now...
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:31
Hello has anyone use Specflow to test Actors
Paweł Bańka
@pmbanka
Sep 28 2017 13:36
hmm, I'm not sure whether the error is not actually in the FindSerializerForType. If the order of the fallbacks was replaced (first try to fetch serializer for Object, then fallback to default from name) that would probably make more sense
Vagif Abilov
@object
Sep 28 2017 13:40
@Tochemey We haven't used SpecFlow but since we are in F#, we are using TickSpec F# library that roughly does the same thing.
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:40
@object Thank you
Vagif Abilov
@object
Sep 28 2017 13:41
We have a few specifications written in Gherkin that we run using TickSpec. And they include actors in Akka.NET
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:42
I am a C# developer. I am still finding my way around f#
Vagif Abilov
@object
Sep 28 2017 13:42
Then you can use SpecFlow.
Here's an example of our spec:
Scenario 1: First time distribution should add new locator
Given File with essence "A/abc_0.txt" has no distribution history
When File with essence "A/abc_0.txt" is distributed with result
| StorageProvider| State | Length |
| Akamai | Completed | 12345 |
Then File with essence "A/abc_0.txt" should have the following distribution locators
| StorageProvider| State | Length |
| Akamai | Completed | 12345 |
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:43
ok
Vagif Abilov
@object
Sep 28 2017 13:44
The good thing about F# is that its function may include spaces, so your implementation names can be identical to Gherkin steps:
let [<Given>] File with essence "(.*)" has no distribution history (fileEssence : string) =
let fileDistribution = getFileDistributionActor fileEssence
fileDistribution <! RemoveLocators
fileDistribution |> waitForMessagesToBeExecuted
But earlier I spent long time with SpecFlow. It's a very sold approach for acceptance tests especially if you want to present the specs to non-technical people.
Paweł Bańka
@pmbanka
Sep 28 2017 13:46
@Tochemey also check if your tooling supports jumping between spec file and the steps implementation, I find myself cursing lack of it every now and then
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:46
@object Can you educate me on how I can use it with the TestKit since I am a C# guy
Paweł Bańka
@pmbanka
Sep 28 2017 13:47
and also, personal opinion, if you don't have a lot of scenarios (and I mean a LOT), then the overhead of those spec files is not really worth it... nicely written and structured unit tests may give you 80% of gains with 20% effort
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:47
Hahaha
Vagif Abilov
@object
Sep 28 2017 13:49
@Tochemey If you want to use TestKit, then most likely you are on unit test level, and I agree with @pmbanka that unit tests may cover lion share of your validation needs.
Our Gherkin specs don't reference Akka TestKit because they are on a higher level - they test higher level scenarios.
Paweł Bańka
@pmbanka
Sep 28 2017 13:50
yeah, they are more like integration tests
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:50
ok
Vagif Abilov
@object
Sep 28 2017 13:51
Bear in mind also that end-to-end tests will use much longer time to execute (and will more often fail due to some external reasons).
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:51
@object Thank you Broz
Vagif Abilov
@object
Sep 28 2017 13:52
This is why Akka TestKit is invaluable for fast in-memory tests - and actors in general are very good fit for in-memory tests. With actors you only need to test that outs are corectetly generated for ins.
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:52
Cool man
Vagif Abilov
@object
Sep 28 2017 13:53
So with TestKit we fake the external dependencies so we can quickly verify that actors do their job and only that.
With Gherkin it's more end-to-end validation that the whole system works.
Arsene T. Gandote
@Tochemey
Sep 28 2017 13:54
I have been using TestKit smoothly. I just wanted to know how best to fit in BDD (if it is feasible)
Vagif Abilov
@object
Sep 28 2017 14:15
I would rather not fit TestKit into BDD. They address different concerns.
Bartosz Sypytkowski
@Horusiath
Sep 28 2017 14:29
@ThmX akka won't connect your bytes into a single message automagicaly, simply because TCP itself doesn't recognize end of one message and start of another one. From TCP perspective it's only pipe for bytes. You can concat messages together using i.e. Framing/JsonFraming (you can see example here - it's not yet part of the documentation, but it's going to be soon).
Regarding 256B limit - akka does buffer pooling in TCP connections, but they are packet into segments chunked into byte buffers. Buffer size is predefined, the part of configuration responsible for it can be found here. It works like that: Buffer pool is allocating memory in segments, then chunks them into buffers - when an incoming TCP packet needs to be handled, it gets a single buffer and returns it after reading. If the number of handled packages is bigger than number of allocated buffers, a new segments is being allocated. This way we don't allocate buffers one by one - it's not efficient - but in larger segments of memory.
Vagif Abilov
@object
Sep 28 2017 14:33
@Horusiath Do you know what SQL batching journal only deals with EventJournal, and not SnapshotStore? In case an application requests state from large number of actors, and many of them have long history with snapshot optimization, won't it cause the same problem with exceeding SQL connection pool limit?
Bartosz Sypytkowski
@Horusiath
Sep 28 2017 14:36
@object I know, unfortunately I haven't got enough time to optimize that.
Vagif Abilov
@object
Sep 28 2017 14:38
I see. Then it confirms our theory. We cleaned up all snapshots and for some time the app was running without errors, then after some snapshots were created we saw growing number of timeout and exceptions because of them.
Bartosz Sypytkowski
@Horusiath
Sep 28 2017 14:38
but afaik recovery mechanics is now token based - it means that Persistence has an arbitrary number of actors that can recover at any single time. When new actor want to recover, it must wait until a token will be given to it, and return it after recovery has finished
Vagif Abilov
@object
Sep 28 2017 14:40
But in case actor state is requested using Ask and timeout is not long, then there will be a problem.
Thx
Bartosz Sypytkowski
@Horusiath
Sep 28 2017 14:41
yes, this is only way to somewhat mitigate the problem - slow (non-batched) snapshotting will still be a problem
Vagif Abilov
@object
Sep 28 2017 14:43
We kind of provoked the problem by eliminating consistent hash routers (remember our chat about this). Now we create actors more aggressively (before they were limited by router configuration), so it's not unlikely we have a thousand actors eagerly trying to read its state.
Bartosz Sypytkowski
@Horusiath
Sep 28 2017 14:48
do they need to be persistent?
Vagif Abilov
@object
Sep 28 2017 14:57
Some of them yes, they need to be persistent.
Although when it becomes a bottleneck we may need to review our persistent actor design and instead of having large number of actors with small persistent state put keep the state in some parent actors thus reducing number of persistent entities.
Aaron Stannard
@Aaronontheweb
Sep 28 2017 16:07
@kariem-ali were you able to resolve this?
Vagif Abilov
@object
Sep 28 2017 16:10
@Aaronontheweb we are having the same issue and @pmbanka has some theory about what's causing it (see his messages above). He may create a PR if he confirms his theory.
Aaron Stannard
@Aaronontheweb
Sep 28 2017 16:12
can we get an issue filed so we can triage it? @object @pmbanka
Vagif Abilov
@object
Sep 28 2017 17:07
Ok, I'll ping Pawel.
Another issue. I wrote here yesterday and later sent pr for Akka.Quartz.Actor that must be updated for Akka.NET 1.3.1, otherwise it fails scheduling jobs with serialization exception. It would be great to release a new Nuget package so Quartz can be used with the latest Akka.
Zetanova
@Zetanova
Sep 28 2017 18:06
hi, if i got corrupted events inside the Event journal (extended events in development), is there a why to patch them?
basicly to overwrite them only once
Zetanova
@Zetanova
Sep 28 2017 18:17
The AR PersistenceId is the AR Id and the AR-Id i would need to reuse
Zetanova
@Zetanova
Sep 28 2017 18:22
the only clean way i can think of is to create a new PersistenceId and persist with it the patched events, but then i would need a mapping between AR-Id and PersistenceId would could get ugly
Zetanova
@Zetanova
Sep 28 2017 18:39
after thinking about it, i got my answere, the PersistenceActor need to go in Migration/Repair state and pushes all events (including the patched ones) into the journal again, after delete all old events from the journal and then leaving the migration/repair state
Zetanova
@Zetanova
Sep 28 2017 19:32
the filter dont realy help, because the events can by views as damaged
and need only to be repaired/migrated once
think to add the migrated-events to the jjournal and then to remove the old once will help
i am thinking to extend the persistence framework with a metalib to store there the event-type + whole binary assembly
if the event-type-definition would change AutoMapper could upcast the events automaticly or to put there a handwriten map, if required (like filter)
Bartosz Sypytkowski
@Horusiath
Sep 28 2017 19:37
isn't this what event adapters do?
Zetanova
@Zetanova
Sep 28 2017 19:38
the main prob is the assembly
one change and its over
.net can load assemblies in multiple versions
so the "MyNamespace.MyEvent, MyAssembly.dll, version 1.0"
and
so the "MyNamespace.MyEvent, MyAssembly.dll, version 1.1"
could be loaded in parallel
AutoMapper can map this, or an upcast routin could do it like
MyNamespace.MyEvent Upcast(dynamic oldEvent) { ... }
if the journal knows about the excact assembly version it could load it from the metastore and trigger an upcast
TNew Upcast<TNew, TOld>(TOld oldEvent) { ... }
Zetanova
@Zetanova
Sep 28 2017 19:45
the metalib needs only to check each event-type on first use and generate a hash out of the typecode. If its registered it will do nothing and if its not if will dump the assembly into its store
the jorunal could save beside the metadata the hash/id of the event
and could allways deserialize the event with the excact event-typecode as serialized
after the journal could look if this type is in the newest version and if not upcast the event to the newest (current assembly)
the result would be that any change of an event-type would not break the deserialization with garanty
the upcast can work and if not can be handled my a "filter"
Zetanova
@Zetanova
Sep 28 2017 19:53
to make it even good performand they whole eventstream of an PersitenceId could be upcasted and persitet to the end of the stream and the old events removed
the metalib could even save the type of the serializer with its options
so that the eventstream could switch the serializer in a future date
even in remoting it would be a help
one site could request the required assembly version and downcast/upcast the message
for the other site
then even systems with the same assemblies but in different versions have no issue
Zetanova
@Zetanova
Sep 28 2017 20:08
the current problems what i see is:
x) extension will break the eventstream most of the time
x) changing the serializer is not possible
x) changing serializer options or updating it can break the eventstream
of course there is the rule of not modefing event-types after first use, but still it happends specially at development time
Bartosz Sypytkowski
@Horusiath
Sep 28 2017 20:21
@Zetanova from v1.3.1 serializerId is the field stored as a column - you may change it if you want, if you're going to use different serializer type. The rest is depending on the specific serializer limitations.
Zetanova
@Zetanova
Sep 28 2017 20:23
So i could make a generic serializer now that could load the old-type from a saved assembly, use common serializer to deserialize the message and upcast it to the current one
that hook i missed in 1.2