These are chat archives for akkadotnet/akka.net

12th
Apr 2018
AndreSteenbergen
@AndreSteenbergen
Apr 12 2018 06:28
@Aaronontheweb I understand, it can really crawl a lot, it seems changing the buffer size fixed the issue, don't know why or how, but the issue is gone, and a waiting list of 4 million pages is already absurd, so if the buffer would be full, I would have used too much memory already, so going back to a few 1000 is goog enough as well.
Aaron Stannard
@Aaronontheweb
Apr 12 2018 06:29
is this the Akka.Remote buffer size or the crawler's stream setup?
because yeah, the Akka.Streams stuff needs to be tuned a bit - I think it's a bit too aggressive in terms of throttling down the crawl rate and level of concurrency at the moment
AndreSteenbergen
@AndreSteenbergen
Apr 12 2018 08:06

Crawler stream set-up, with the smaller buffer I don't seem to get any issues.

I still need to tweak the config somewhat, on all node the standard config creates 2 download actors (per domain as I see it), with the maxConcurrency, making it distributed over multiple nodes will look a ddos attack in no time, don't want to bring down sites.

nathvi
@nathvi
Apr 12 2018 15:55
I have a question with Akka Remote Deployment. If I deploy actor A from process A to process B, and process B stops or gets terminated, where does the state of actor A reside?
Aaron Stannard
@Aaronontheweb
Apr 12 2018 16:54
actor A lives physically in process B
so actor A's state will be destroyed if B is terminated
you'd want to persist that state to Akka.Persistence or replicate it elsewhere
to make it recoverable or otherwise available
Lutando Ngqakaza
@Lutando
Apr 12 2018 17:27
when doing cross cluster messaging do my messages have to be mutable?
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:27
nope
just need to be serializable
they should always be immutable ideally
Lutando Ngqakaza
@Lutando
Apr 12 2018 17:28
yeah
I am trying to figure out why a EventStream.Publish object is coming back with null fields where those fields are pocos with private setters
everything else (mutatable fields) are fine
if i make them private set they appear null too
will dig further
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:29
using the built-in JSON serializer I take it?
Lutando Ngqakaza
@Lutando
Apr 12 2018 17:29
i beleive we are using the defaults for serialization
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:29
ok, got it
the JSON serializer just may not be able to figure out how to populate those values since it relies on conventions / magic IMHO to do it
but imho
Lutando Ngqakaza
@Lutando
Apr 12 2018 17:30
oh so between clusters it uses JSON and not Hyperion as default?
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:30
yeah Hyperion isn't used as a default anywhere at the moment
if you're serious about using Akka.Remote or Akka.Cluster in production though
you're better off going with hand-rolled serialization using something that has an explicit scheme like Google.Protobuf
better performance, first and foremost
but more importantly, this type of issue doesn't happen
since you can explicitly control serialization
also makes it easier to version messages on the wire too
Lutando Ngqakaza
@Lutando
Apr 12 2018 17:31
we will probably end up there, right now were doing a "large" proof of concept
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:31
we ship with JSON.NET / Hyperion in the future as a default because it's easy to get up and running
but those polymorphic serializers aren't perfect
might have to give them some hints on how to do their job
Lutando Ngqakaza
@Lutando
Apr 12 2018 17:32
i see
Jesse Connor
@jesseconnr
Apr 12 2018 17:46
I'm having an issue with message delivery in a setup remoting between a windows service and an asp.net site. I'm using an aggreggator to send a large amount of messages to keep the size small, but out of 3500 messages only 1700 or so are making it. In smaller quantities, when there's only a few hundred there's no problems.
My agreggator and the hocon config for the two is here: https://gist.github.com/jesseconnr/256da997b49e38026c32f4bc2987adf5
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:47
are you hitting your receive timeout?
and shutting down?
Jesse Connor
@jesseconnr
Apr 12 2018 17:48
Both are on the same machine. I get errors like this Error while decoding incoming Akka PDU of length 28885 but I'm not sure what to make of it.
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:48
ah shit
that's a serialization problem
Jesse Connor
@jesseconnr
Apr 12 2018 17:48
Oh the 30 seconds? No I don't think so.
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:48
this is with Hyperion
do you get that issue without it?
and is this on Linux?
Jesse Connor
@jesseconnr
Apr 12 2018 17:48
It's on windows.
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:49
so we've had reports of DotNetty fucking up its length-frame encoding
on v0.4.6
originally the reports were Windows-only
err
Linux only
Jesse Connor
@jesseconnr
Apr 12 2018 17:49
I send the messages in the same order and tried to figure out if it was a particular one that was having a serialization issue, but the point at which the error occurs seems random.
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:49
had one last week on Windows
no no, this isn't something you're doing wrong
it's DotNetty
Jesse Connor
@jesseconnr
Apr 12 2018 17:50
I believe I read that issue, but I think I'm on 0.4.7
Yep, as far as I can tell, DotNetty is all 0.4.7
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:51
v0.4.7 has major issues
like, their API is outright broken there
I've literally never had this issue happen to me on v0.4.6 but other users have reported it
I would strongly, strongly encourage rolling back to v0.4.6
Jesse Connor
@jesseconnr
Apr 12 2018 17:52
I was having the issue on 0.4.6 and an older version of akka and upgraded to see if it would fix it and no dice.
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:52
well shucks
can you guys do me a favor and open an issue on the DotNetty github
I don't have a window where I can even look into the issue for another 5 days
you're going to get a faster resolution on it if we can get their org to start looking at the problem
this is not an Akka.NET issue
it's a problem with length-frame encoding not being written or read correctly
Jesse Connor
@jesseconnr
Apr 12 2018 17:54
Ah yeah I see the bug report and the mention. I get that same error. DotNetty.Codecs.TooLongFrameException: Adjusted frame length exceeds 128000: 419561476 - discarded
Aaron Stannard
@Aaronontheweb
Apr 12 2018 17:55
Azure/DotNetty#360
Jesse Connor
@jesseconnr
Apr 12 2018 17:58
hmm, v5 was next week in february, but I guess it never got there
Aaron Stannard
@Aaronontheweb
Apr 12 2018 18:12
I have the DotNetty guys on the horn in their gitter now
please go post a copy of your error message et al
that would help
have to demonstrate the pervasiveness of the issue
Jesse Connor
@jesseconnr
Apr 12 2018 18:23
Would having an old 1.3.0 version of Akka.Serialization.Hyperion possibly have an effect?
I just realized that when everything else was updated, that wasn't because I didn't check the prerelease in visual studio.
Aaron Stannard
@Aaronontheweb
Apr 12 2018 18:24
try an upgrade and see what happens
but this looks really clearly like a DotNetty issue
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 18:24
Can you use a Lighthouse type of solution with Akka.Cluster.Sharding?
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 18:26
ok let me try that
thanks
Jesse Connor
@jesseconnr
Apr 12 2018 18:33
Ah well, I would try, but using nuget from within visual studio to downgrade was apparently a bad idea, everything is wonderfully jacked now. :)
Aaron Stannard
@Aaronontheweb
Apr 12 2018 18:56
lol
well, got some answers from the DotNetty crew that make some sense
due to some issues with .NET Core, it's possible that the byte buffer pools DotNetty uses can be released early
by the socket itself
especially true on Linux
we use the default byte buffer allocator in our DotNetty transport, which uses pooling
this normally helps performance big time, because it eliminates allocations and keeps GC pressure down
guess I could expose some sort of setting to change the pooling strategy within Akka.Remote's DotNetty transport ("none" being an option)
but anyway, that issue is fixed in DotNetty v0.5.0 and they have one last PR they're trying to sort out before moving ahead with that
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 19:12
@Aaronontheweb thanks for the help that did it. now onto the next problem.
If I have 2+ nodes (cluster sharding) running, and I start sending messages, none are getting processed, once I drop down to 1, processing starts on that node, then I can later add additional nodes and they will sometimes pick up the load.. Is this by design?
am I missing another config?
Aaron Stannard
@Aaronontheweb
Apr 12 2018 19:15
oh man, this is running v1.3.5 right?
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 19:15
yea
beta60
Aaron Stannard
@Aaronontheweb
Apr 12 2018 19:15
sounds like this bug we fixed
that we're trying to ship in 1.3.6
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 19:15
ok
Aaron Stannard
@Aaronontheweb
Apr 12 2018 19:15
if you try the nightlies that should work
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 19:15
ok
my bad on the delay on the 1.3.6 release
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 19:16
I understand
when do you think that the cluster sharding stuff will be out of beta?
Aaron Stannard
@Aaronontheweb
Apr 12 2018 19:20
soon, hopefully - we just fixed the biggest bug with it in 1.3.6
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 19:20
ok cool
Jesse Connor
@jesseconnr
Apr 12 2018 19:27
Ah well it's good an update is coming. Almost got this downgrade worked out, hopefully it works, otherwise it seems my only option is going to be custom compiling dependencies to get this app live.
Awesome, I don't know why it didn't work before my upgrade, perhaps cleaning everything out and reinstalling the old version fixed something. Every message came through near instantly on 0.4.6.
Aaron Stannard
@Aaronontheweb
Apr 12 2018 19:31
:+1:
glad to hear it
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 19:35
after moving to nightlys, I am getting this error:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.MissingMethodException: Method not found: 'Void Hyperion.SerializerOptions..ctor(Boolean, Boolean, System.Collections.Generic.IEnumerable1<Hyperion.Surrogate>, System.Collections.Generic.IEnumerable1<Hyperion.SerializerFactories.ValueSerializerFactory>, System.Collections.Generic.IEnumerable`1<System.Type>)'.
Aaron Stannard
@Aaronontheweb
Apr 12 2018 19:36
gotta upgrade the version of the Hyperion serializer too
I suspect
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 19:36
<PackageReference Include="Hyperion" Version="0.9.8" />
Aaron Stannard
@Aaronontheweb
Apr 12 2018 19:36
Akka.Serialization.Hyperion
^^^
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 19:36
ok
I didnt have a reference to it at all
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 20:04
@Aaronontheweb ok nightly builds gave me much better results
thanks again for the help
one question, what triggers a rebalance?
Ryan Anthony
@ryandanthony_twitter
Apr 12 2018 20:34
never mind, it looks like I just didnt have enough unique shardsIds in the dataset I was using.