These are chat archives for akkadotnet/akka.net

30th
Jun 2017
Jesse Connor
@jesseconnr
Jun 30 2017 00:50
No? Maybe it's just me, I like to see lots of theoretical models before settling.
And it's really a slow process using things like draw.io.
Jesse Connor
@jesseconnr
Jun 30 2017 00:59
How do you guys handle filling a list of items in a view? Ex. a user visits a page and you need to show them a list of 100 cars. Each car is its own actor with various parts, etc.
The later parts are easy, when someone edits a car an event gets emitted to the viewers of the car.
Hydrating that list isn't though, I could loop through all car actors and build a list when a user needs the list and try to pass the big last back down the pipe.
Gregorius Soedharmo
@Arkatufus
Jun 30 2017 01:00
ah, you mean as a visual design aid
Jesse Connor
@jesseconnr
Jun 30 2017 01:01
or I could loop through them all and fire them down the pipe individually using websockets
Gregorius Soedharmo
@Arkatufus
Jun 30 2017 01:03
or have it paged so you dont have to send 100 data all at once?
that would need coordinating though
Jesse Connor
@jesseconnr
Jun 30 2017 01:05
that could work, wouldn't be too hard having a max set on a quantity to return
it would create more requests as the page is viewed and then subsequently filters are put in place
but I suppose that doesn't really matter
Gregorius Soedharmo
@Arkatufus
Jun 30 2017 01:07
well, its a trade off between bandwith and request
Jesse Connor
@jesseconnr
Jun 30 2017 01:08
I'm thinking I might hold state in a manger in a dictionary with key being the id, and a tuple for the value with actor reference and object
bah, actually that probably wouldn't work
Gregorius Soedharmo
@Arkatufus
Jun 30 2017 01:09
it might, make the manager itself an actor and make it an aggregator
state manager, not actor manager
i guess it can be an actor manager too
Jesse Connor
@jesseconnr
Jun 30 2017 01:19
that's a good idea and gave me another one as well
so my initial thought is, user views page, kicks off initial request with a max request #, the message does its thing, in the end the items get pushed back down to the user with a X/10 in the message so things aren't populating on the page 1 at a time
and then I can just keep it flowing, obviously not streams, but it means I don't need to bother with asking for large lists of data
Arsene Tochemey GANDOTE
@Tochemey
Jun 30 2017 10:48
Hello Gentlemen. So far all the tutorial I have seen on Akka Streams are scala related. Is there any one related to AKKA.NET?
So far I can figure out the Scala one but I find a bit difficult to translate it to c#.
Andy Kutruff
@akutruff
Jun 30 2017 13:08

@Aaronontheweb

works pretty well - only real catch is you have to handle:

  1. Serialization
  2. Partial reads
  3. Message framing
Yikes... that's a big chunk.... I was hoping using untrusted mode was going to get a client/server done.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 13:10
untrusted mode in Akka.Remote will definitely take care of the most dangerous parts
since it disallows the connected parties from doing anything like starting / stopping actors across network boundaries
but that code for Akka.IO isn't so bad
Andy Kutruff
@akutruff
Jun 30 2017 13:11
Yeah, I figured I'd have to have a central gateway actor that handles auth on the gameservers
Aaron Stannard
@Aaronontheweb
Jun 30 2017 13:11
pulling up a piece from a private project
Andy Kutruff
@akutruff
Jun 30 2017 13:12
I'm really scrambling to see if I can listen on a port on iOS regardless
I like the decoupling nature of the peer to peer networking.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 13:13
internal sealed class MessageEncoder
    {
        private ByteString _remainder = ByteString.Empty;

        /// <summary>
        ///     Frame a single message
        /// </summary>
        /// <param name="bytes">The raw message bytes.</param>
        /// <returns>A bytestring with length frame headers.</returns>
        public ByteString Encode(byte[] bytes)
        {
            var b = new ByteStringBuilder();
            b.PutInt(bytes.Length, ByteOrder.LittleEndian);
            b.PutBytes(bytes);
            return b.Result();
        }

        /// <summary>
        ///     Decode an incoming message and produce 0 or more messages depending on header values.
        /// </summary>
        /// <param name="byteString">The incoming data.</param>
        /// <returns>0 or more individually framed messages.</returns>
        public IEnumerable<byte[]> Decode(ByteString byteString)
        {
            var working = byteString;
            if (_remainder.NonEmpty)
                working = _remainder.Concat(byteString);

            var i = working.Iterator();
            var output = new List<byte[]>();
            while (i.HasNext)
            {
                var messageLength = i.GetInt(ByteOrder.LittleEndian);
                if (i.Len >= messageLength)
                {
                    output.Add(i.GetBytes(messageLength));
                }
                else
                {
                    _remainder = new ByteStringBuilder()
                        .PutInt(messageLength, ByteOrder.LittleEndian)
                        .Append(i.ToArray()).Result();
                    break;
                }
            }

            return output;
        }
    }
Andy Kutruff
@akutruff
Jun 30 2017 13:13
As it makes sense to run the same code for a single player game locally, and a multiplayer game remotely
Aaron Stannard
@Aaronontheweb
Jun 30 2017 13:13
frame-length encoding with partial-message handling for Akka.IO
uses a 4-byte header to describe the length of the payload
I basically stick this and my serializer that consumes the byte[] array inside the actor who receives the Tcp.Received events from Akka.IO
Andy Kutruff
@akutruff
Jun 30 2017 13:14
I do something similar for our existing stuff.
So Akka.IO will make the actors connection centric
_channel.Tell() rather than _actor.Tell()
Aaron Stannard
@Aaronontheweb
Jun 30 2017 13:16
it's still actor.tell
 public MiniClient(IActorRef testActor)
        {
            _testActor = testActor;

            Receive<Tcp.Connected>(c =>
            {
                _connection = Sender;
                _connection.Tell(new Tcp.Register(Self));
                _testActor.Tell("connected");

                Stash.UnstashAll();
                Become(Connected);
            });

            ReceiveAny(_ => Stash.Stash());
        }
Andy Kutruff
@akutruff
Jun 30 2017 13:16
right, right. I'm speaking architecturally
Aaron Stannard
@Aaronontheweb
Jun 30 2017 13:16
so this is an actor I use for testing, but you have some specific connection events you handle as messages
Tcp.Connected means you're now connected to some host
Andy Kutruff
@akutruff
Jun 30 2017 13:16
Yeah, this is all easy peasy
Aaron Stannard
@Aaronontheweb
Jun 30 2017 13:17
you reply back with a Register command.... this tells Akka.IO which actor is going to be handling the data on the socket
private void Connected()
        {
            Receive<SendCommand>(send =>
            {
                var serialized = Encoder.Encode(Serializer.Serialize(send.Command));
                _connection.Tell(Tcp.Write.Create(serialized));
            });

            Receive<Tcp.CommandFailed>(failed => failed.Cmd is Tcp.Write, failed => { _testActor.Tell(failed); });

            Receive<Tcp.Received>(r =>
            {
                try
                {
                    var bytes = Encoder.Decode(r.Data);
                    _log.Debug("Received [{0} bytes] from host", r.Data.Count);
                    foreach (var b in bytes)
                    {
                        var deserialized = Serializer.Deserialize(b);
                        _log.Debug("Deserialized [{0}] from host", deserialized);
                        _testActor.Tell(deserialized);
                    }
                }
                catch (Exception e)
                {
                    var ex = e;
                }
            });

            Receive<Tcp.Closed>(c =>
            {
                _testActor.Tell("finished");
                Context.Stop(Self);
            });

            Receive<string>(str => str.Equals("close"), s => { _connection.Tell(Tcp.Close.Instance); });
        }
this is using the same encoder from earlier
so Akka.IO basically abstracts over the socket and turns it into "actors with bytes"
this will allow you to avoid having an inbound listening port on your iOS devices
Akka.Remote is more abstracted than this in that it handles all of the message serialization et al, but yeah, it's designed for building peer-to-peer systems
hence it opens an inbound port for every participant
Andy Kutruff
@akutruff
Jun 30 2017 13:22
Yeah, which is rad. If I'm having to couple my game code to the concept of a connection again, there is sadness. Like right now, our systems have a "local memory" connection and a tcp connection to achieve some level of location transparency. The actual game logic interacts through the channel which forces a separation between client and server code.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 13:23
welp, if you can get past the inbound port thing
you should be able to achieve that
Aaron Stannard
@Aaronontheweb
Jun 30 2017 13:27
but you'll eventually arrrive at https://www.youtube.com/watch?v=uwmeH6Rnj2E
Andy Kutruff
@akutruff
Jun 30 2017 13:28
hhaha, jesus this is epic
Jesse Connor
@jesseconnr
Jun 30 2017 13:52
If you create a child actor using an object, then the child actor maintains state on that object, when it restarts the object is rebuilt so the restarted child actor would go back to the original state? Actually, I guess not unless it copies the object instead of just using the reference given.
Which I guess depends on how Props actually work.
Andy Kutruff
@akutruff
Jun 30 2017 14:21
Just a status update on IL2PP / AOT / Akka.Remote - errors are getting swallowed on a background thread or there is a hang due to the networking stack or maybe firewall/port forwarding. Trying to figure out how to get unhandled exceptions off the background threads. However, I need to look at Akka.IO now that Aaron sent that cat video
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:21
@akutruff do you have a means of extracting the Akka.NET logs from an AOT-compiled app?
all of that stuff should be captured in the error logs
Andy Kutruff
@akutruff
Jun 30 2017 14:23
Not sure. Would need it to go to a file instead of standard I/O, or I need to pipe it into unity's Debug.Log() with a custom logger. However, I fear using Debug.Log() on a background thread with any Unity API. On top of that, I'm not sure what sandboxing is happening with this universal windows platform stuff.
Also, I should probably figure out how to increase the log level up from info
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:24
akka.loglevel = DEBUG | INFO | WARNING | ERROR
in HOCON configuration
ActorSystem.Create("MySys", ConfigurationFactory.ParseString("akka.loglevel = DEBUG"))
Andy Kutruff
@akutruff
Jun 30 2017 14:25
Perfect. If I need to specifiy a custom logger, is that where I do it as well?
Like UnityDebugLogger
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:26
https://github.com/akkadotnet/Akka.Logger.Serilog - here's an example of a custom logging implementation
typically it's just one class
but to load that up....
Jesse Connor
@jesseconnr
Jun 30 2017 14:26
loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:26
yeah
what @jesseconnr said
Andy Kutruff
@akutruff
Jun 30 2017 14:26
Also, I verified that IL2CPP is failing to properly convert asyncs with exception filters at compile time. Rewriting the catch blocks with if statements let it go through.
cool, thanks.
Jesse Connor
@jesseconnr
Jun 30 2017 14:27
nlog + log4view is a nice combo
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:27
s what you need to implement
Andy Kutruff
@akutruff
Jun 30 2017 14:29
As I'm poking around though... I see a lot of allocation sprinkled around which may be a roadblock for Unity / Games use if people need to do actor operations per frame. A GC pause that causes a frame hiccup unfortunately is a min business blocker
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:30
which area?
the actor mailbox?
Andy Kutruff
@akutruff
Jun 30 2017 14:31
Haven't gotten there. More in actor creation. and lots of API's using params
IEnumerables rather than List<> or arrays.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:31
actor creation is definitely going to have some allocations
I have no idea what the performance characteristics of the actors will be after IL2CPP
Andy Kutruff
@akutruff
Jun 30 2017 14:32
Aggressive object pooling is usually the way I go, but at the cost of cache coherency and memory fragmentation
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:32
but running on a normal .NET CLR box without async GC turned on
Andy Kutruff
@akutruff
Jun 30 2017 14:32
I assume the perf will be the same-ish regardless of IL2CPP
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:32
D2 V2 VM
actors can process about 4.5m messages per second on average
Andy Kutruff
@akutruff
Jun 30 2017 14:33
At the end of the day, it's just about using managed code
D2 V2 VM?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:33
that's the generation of azure VM we run on
modern CPU
2 logical cpus
Andy Kutruff
@akutruff
Jun 30 2017 14:33
Oh, I'm AWS versed, not Azure
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:33
which.... if I understand Azure pricing... is probably 1 physical core with hyper-threading
we run our benchmarks and tests on under-powered machines on purpose
Andy Kutruff
@akutruff
Jun 30 2017 14:34
Ah, but do you have any GC pauses? I assume you're running background GC too
The pauses are the thing here, not throughput on average
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:34
we don't run background GC on the benchmarks
because it becomes really tough to measure GC when it's async
(in a benchmark, not when you're profiling over a long period of time)
let me grab the latest...
Andy Kutruff
@akutruff
Jun 30 2017 14:36
I'm not sure what GC Unity is using right now either. Could be old Mono, new Mono, or Core
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:36
# Akka.Tests.Performance.Actor.ActorThroughputSpec+ReceiveActor_Throughput
__Measures the throughput of an ReceiveActor__
_6/29/2017 7:53:45 PM_
### System Info
```ini
NBench=NBench, Version=1.0.1.0, Culture=neutral, PublicKeyToken=null
OS=Microsoft Windows NT 6.2.9200.0
ProcessorCount=2
CLR=4.0.30319.42000,IsMono=False,MaxGcGeneration=2

NBench Settings

RunMode=Iterations, TestMode=Measurement
NumberOfIterations=13, MaximumRunTime=00:00:01
Concurrent=True
Tracing=True

Data


Totals

Metric Units Max Average Min StdDev
TotalCollections [Gen0] collections 176.00 163.08 154.00 7.12
TotalCollections [Gen1] collections 77.00 71.23 66.00 3.79
TotalCollections [Gen2] collections 23.00 10.38 1.00 6.69
[Counter] MessageReceived operations 30,000,000.00 30,000,000.00 30,000,000.00 0.00

Per-second Totals

Metric Units / s Max / s Average / s Min / s StdDev / s
TotalCollections [Gen0] collections 23.85 21.57 17.99 1.89
TotalCollections [Gen1] collections 11.13 9.46 7.13 1.23
TotalCollections [Gen2] collections 2.69 1.30 0.15 0.75
[Counter] MessageReceived operations 4,646,053.50 3,982,523.24 3,194,072.82 460,691.27
```
Andy Kutruff
@akutruff
Jun 30 2017 14:36
Either way, latency is the thing here. Could be doing 1M messages a sec, but if the GC pauses the threads for 5ms, a frame will drop
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:37
so that's total figures for a receive actor processing 30 million messages
from the last time we merged something into the v1.3 branch
Andy Kutruff
@akutruff
Jun 30 2017 14:37
Is this steady state, or does it include warmup ?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:37
we warm it up first
although the warm-up's aren't as effective here
since it's running multi-core
can't be as precise about things like cache-pinning
Andy Kutruff
@akutruff
Jun 30 2017 14:38
Wait, 23.85 Gen0's per second?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:38
yep
Andy Kutruff
@akutruff
Jun 30 2017 14:39
yikes...
In server environments, these numbers are super great
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:39
I think the allocations there are probably coming from the mailbox
uses a ConcurrentQueue under the hood
Andy Kutruff
@akutruff
Jun 30 2017 14:39
Well, the messages themselves
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:39
and it has to allocate new array segments et al
Andy Kutruff
@akutruff
Jun 30 2017 14:39
That's usually the most chatty
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:39
yep, and the messages themselves
Andy Kutruff
@akutruff
Jun 30 2017 14:40
I object pool them in our stuff, and just eat the copy cost and eat the fact that we lose constructors
that we use for collecting this data
Andy Kutruff
@akutruff
Jun 30 2017 14:42
I can likely adapt a couple of these for Unity scenarios if I can get that far.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:43
so we're re-using the same message over and over there
in that benchmark
so the allocations are probably coming from some of the plumbing
we use value types for the message envelope
so that won't show up in the GC result
Andy Kutruff
@akutruff
Jun 30 2017 14:43
There's boxing there...
obj is int
used the typed receives
we have a few different actor implementations we test
actor performance is mostly bounded by the throughput of the dispatcher
Andy Kutruff
@akutruff
Jun 30 2017 14:45
With Receive<int> still box?
or do you have a type erased envelope?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:45
it's probably still going to box
the envelope doesn't describe the message type at all
Andy Kutruff
@akutruff
Jun 30 2017 14:46
Oh, you're using an envelope otherwise. I was just guessing at your dispatch table implementation
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:46
.... actually shit, that might be a good idea
hadn't thought of that
Andy Kutruff
@akutruff
Jun 30 2017 14:46
Yeah, it's a good trick for these things
Envelope<T> : Envelope
It's a bitch though if you introduce any form of converters or unwrappers though
like List<T>
It's how I avoid boxing for serialization yet still have a pluggable type serializers that support value types without boxing
(and avoiding using reflection.emit)
this is the execution engine
it's abstract enough to allow for things like running actors on the built-in thread pool, the custom thread pool, or even the UI thread
Andy Kutruff
@akutruff
Jun 30 2017 14:49
Alrighty, well, the good thing from looking at the code is that as long as I can actually use this actor model for the business logic, I see all the micro optimizations being possible, which is great
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:49
I'm wondering if we're being allocate-y in there...
Andy Kutruff
@akutruff
Jun 30 2017 14:49
Can leave that for later in the cycle or as I go along. As long as the Akka project is amenable to making that a priority
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:50
we love optimizations
hence why we benchmark aggressively
Andy Kutruff
@akutruff
Jun 30 2017 14:50
Well, if there is any domain that requires optimization all the way through, it's games
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:51
I'm sure our users who work in finance, energy, manufacturing et al would be more than happy to get some of that type of performance in their apps too :)
Andy Kutruff
@akutruff
Jun 30 2017 14:51
However, I do know a bunch of Wall St. people who are in love with JVM Akka, which is an entirely different can of worms, which is why I know it's a good tech
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:51
I need to turn on GC profiling for our dispatcher specs
can't believe we don't have that running right now
Andy Kutruff
@akutruff
Jun 30 2017 14:52
Oh, I see that it's there already. I used ot work in finance, and I'd already thumbs up what I see here for that domain
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:52
I'd love to see this stuff running on a Unity client
Andy Kutruff
@akutruff
Jun 30 2017 14:52
Have you seen Stack Overflow's post on their issues with GC pauses?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:52
I remember reading one a while back
btw, if you're willing to trade some throughput in exchange for lower GC
you can swap in stuff like custom mailboxes and dispatchers
where you use object pooling much more aggressively and protect the critical region with a lock
rather than do the CAS type stuff that we do
Andy Kutruff
@akutruff
Jun 30 2017 14:54
I was peeking at the dispatchers, because I likely need to run the front-end single threaded for the game logic, but keep the network on the background.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:54
the SynchronizedDispatcher will bind to the synchronization context of whomever called ActorOf
Andy Kutruff
@akutruff
Jun 30 2017 14:54
Well, problem with pooling is that unless you have copy semantics, it becomes untenable. Message passing has immutability which affords copy semantics
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:55
so if you do a ActorSystem.ActorOf command on the UI thread to spawn that actor, it can effectively run on it
Andy Kutruff
@akutruff
Jun 30 2017 14:55
Otherwise, you end up writing your own GC, or ref counting (yuck)
I'd like to make that an ambient thing, so I don't have to couple my actor creation to knowing where it's executing. Possible?
Like SynchronizationContext
Aaron Stannard
@Aaronontheweb
Jun 30 2017 14:57
if you don't need to run on the UI thread
you can create a ForkJoinDistpacher
with a thread pool size of 1
and have all of the game logic actors run on that
Andy Kutruff
@akutruff
Jun 30 2017 14:59
Well, I need the game logic actors to be on the UI thread. I want to have an event stream observer or a hook into the persistent actors so that my game visuals can observe the actors and react per event
Also, I want those actors actually to be a mirror of an identical set of actors that may or may not be ahead of the UI actors
that way, I can control the message pace of the game actors so that it's visually appealing, but not blocking the latest up to date actors that are talking to the network
Two simulation spaces - one that's a "lie" to hide latency and jitter for the user to control and be lively. another that is the "truth" I reconcile the lie and truth as they deviate
The truth can be on the threadpool ideally
But I need to be able to pump the messages to the UI thread actors
Which... actor cloning
(with current state)
Andy Kutruff
@akutruff
Jun 30 2017 15:27
@Aaronontheweb for the loggers =[""Chat.UnityLogger""] do I need to actually create the actor, or does this instantiate one for me on my behalf?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 15:27
you need to create a logging actor impl for that
but if you have the implementation
then Akka.NET will start the actor for you automatically
Andy Kutruff
@akutruff
Jun 30 2017 15:27
I made one from the Serilogger
Hmm
Aaron Stannard
@Aaronontheweb
Jun 30 2017 15:28
and the type name needs to be fully qualified
in order to be loaded
Andy Kutruff
@akutruff
Jun 30 2017 15:28
ConfigurationException: Logger specified in config cannot be found: "Chat.UnityLogger" Akka.Event.LoggingBus.StartDefaultLoggers (Akka.Actor.Internal.ActorSystemImpl system) (at C:/code/akka.net/src/core/Akka/Event/LoggingBus.cs:100) Akka.Actor.LocalActorRefProvider.Init (Akka.Actor.Internal.ActorSystemImpl system) (at C:/code/akka.net/src/core/Akka/Actor/ActorRefProvider.cs:374) Akka.Remote.RemoteActorRefProvider.Init (Akka.Actor.Internal.ActorSystemImpl system) (at C:/code/akka.net/src/core/Akka.Remote/RemoteActorRefProvider.cs:173) Akka.Actor.Internal.ActorSystemImpl.Start () (at C:/code/akka.net/src/core/Akka/Actor/Internal/ActorSystemImpl.cs:185)
Aaron Stannard
@Aaronontheweb
Jun 30 2017 15:28
yeah, you just need to make it assembly-qualified
Andy Kutruff
@akutruff
Jun 30 2017 15:28
oh, so Chat.Chat.
Oh, Chat.UnityLogger, Chat duh
Aaron Stannard
@Aaronontheweb
Jun 30 2017 15:48
@akutruff did that work
and as an aside.... I'm getting the itch to play some performance golf again
after talking about all of this
going to start by turning on GC profiling for the dispatcher
Andy Kutruff
@akutruff
Jun 30 2017 15:49
Yeah, but now some weirdness... unity loads and unloads a sandbox when you enter "play mode" after exiting play mode, I was still getting log spam
Aaron Stannard
@Aaronontheweb
Jun 30 2017 15:49
I think I see some opportunities to employ some object pooling there
Andy Kutruff
@akutruff
Jun 30 2017 15:49
Which tells me something is leaking across play mode
And that's bad
Aaron Stannard
@Aaronontheweb
Jun 30 2017 15:50
are you shutting down the ActorSystem?
if not, that would do it
Andy Kutruff
@akutruff
Jun 30 2017 15:50
Under IL2CPP though, the logger is failing with a "Task canceled"
Aaron Stannard
@Aaronontheweb
Jun 30 2017 15:50
hmmm
make sure you send back that reply on the "LoggerStarted" message or whatever
Andy Kutruff
@akutruff
Jun 30 2017 15:51
Even if I were/weren't, Unity should be isolating all code loaded
Aaron Stannard
@Aaronontheweb
Jun 30 2017 15:51
so that's a Unity issue?
Andy Kutruff
@akutruff
Jun 30 2017 15:52
Yeah, could be. I hacked putting in a static delegate callback to unity's debug call, but even then that shouldn't be kept in memory by unity. state should be absolutely fresh. Otherwise, there is a buffer that is emptying perhaps
Regardless, IL2CPP isn't working. Might be a timing issue, since starting the app takes a second
Andy Kutruff
@akutruff
Jun 30 2017 15:58
Wow, yeah, Unity does not shutdown app created threads in the Editor... how I have not gotten a bug from this over the years, I do not know.
Andy Kutruff
@akutruff
Jun 30 2017 16:31
I'm disposing the system, but it appears to keep hanging on to some thread or resource until the server console app also shutsdown.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 16:31
is this with Akka.Remote?
Andy Kutruff
@akutruff
Jun 30 2017 16:31
Yeah
Figured I'd take that to conclusion before trying IO
Since IL2CPP is behaving differently than the Unity Editor
I'm using cancellation tokens for the background task though... I may not be doing it right, but I verified that the dispose is being called
Shutting down remote daemon.
This is what gets logged client side when I eventually stop the server console process
Association with remote system akka.tcp://MyServer@localhost:8081 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
So there's defintely something hanging out on the front end
Aaron Stannard
@Aaronontheweb
Jun 30 2017 16:34
I'll look at it - we've had some reports of stuff maybe not shutting down cleanly since we switched to DotNetty in 1.2
Andy Kutruff
@akutruff
Jun 30 2017 16:35
If they are not heartbeating, then you can end up with half open
zombie connection and will hang forever
Aaron Stannard
@Aaronontheweb
Jun 30 2017 16:36
yeah my guess is that the disassocation doesn't terminate outbound connections properly
Andy Kutruff
@akutruff
Jun 30 2017 16:36
Unless it's elevated to the app layer of the protocol half open is unavoidable and it's usually at process termination. Happens even during graceful shutdown.
It's why a lot of load balancers put in timeouts - most apps don't handle it properly
Vagif Abilov
@object
Jun 30 2017 16:37
@akutruff we just switch to DotNetty and have seen similar behavior with Akka.Remote. Didn't investigate yet.
Andy Kutruff
@akutruff
Jun 30 2017 16:39
zombie, half open. If they aren't framing for Akka, then akka will need to heartbeat
which, I'd rather akka heartbeat rather than a lower layer
Aaron Stannard
@Aaronontheweb
Jun 30 2017 16:42
akka heartbeats
Andy Kutruff
@akutruff
Jun 30 2017 16:43
It's usually easy if you have an actor dealing with your connection. You just need to set a timeout on one or both ends. If the timeout expires, you send a ping/pong to verify the other guy is still there.
When you send an actual message, you just reset the timer
for a lively connection, you end up never sending a heartbeat. A high threshold like 15 seconds should do it. AWS defaults their load balancers to timeout idle TCP connections at 30 seconds anyway
This is all assuming that you leave the socket open.
and don't open/close a socket with every message
Oh, and you would only need to do this per endpoint, not per actor
if you're sharing a single socket for all actor communication, the odds of you ever needing to send a heartbeat approaches zero
Andy Kutruff
@akutruff
Jun 30 2017 16:49
Gotta be careful about your timers though - can be heavy
the overhead of calling Timer.Reset() on every message vs. just letting a callback poll without a reset is a consideration
I'm assuming the akka scheduling has some optimization for timers. Haven't gotten there yet
I think this guy's SO answer is decent on the subject
Andy Kutruff
@akutruff
Jun 30 2017 17:03
Oh, I took that as you asking what I meant... Instead you just simply said that you already heartbeat. I'm an idiot
Andy Kutruff
@akutruff
Jun 30 2017 17:11
Akka.Configuration.ConfigurationException: Logger [Chat.UnityLogger, Chat] specified in config cannot be loaded: System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   --- End of inner exception stack trace ---
  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task`1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task`1[TResult].get_Result () [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Event.LoggingBus.AddLogger (Akka.Actor.Internal.ActorSystemImpl system, System.Type loggerType, Akka.Event.LogLevel logLevel, System.String loggingBusName, System.TimeSpan timeout) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Event.LoggingBus.StartDefaultLoggers (Akka.Actor.Internal.ActorSystemImpl system) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.LocalActorRefProvider.Init (Akka.Actor.Internal.ActorSystemImpl system) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Remote.RemoteActorRefProvider.Init (Akka.Actor.Internal.ActorSystemImpl system) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.Internal.ActorSystemImpl.Start () [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.ActorSystem.CreateAndStartSystem (System.String name, Akka.Configuration.Config withFallback) [0x00000] in <00000000000000000000000000000000>:0 
  at Chat.ChatClientProgram+<RunTheChatClient>d__10.MoveNext () [0x00000] in <00000000000000000000000000000000>:0 
  at Chat.ChatClientProgram.RunTheChatClient (Akka.Configuration.Config config) [0x00000] in <00000000000000000000000000000000>:0 
  at Chat.ChatClientProgram+<>c__DisplayClass6_0.<RunRemote>b__0 () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Func`1[TResult].Invoke () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task.ExecutionContextCallback (System.Object obj) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Action`1[T].Invoke (T obj) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task& currentTaskSlot) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task.ExecuteEntry (System.Boolean bPreventDoubleExecution) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <00000000000000000000000000000000>:0 
---> (Inner Exception #0) System.Threading.Tasks.TaskCanceledException: A task was canceled.<---
 ---> System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   --- End of inner exception stack trace ---
  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task`1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.Task`1[TResult].get_Result () [0x00000] in <000000000000000000000000
Andy Kutruff
@akutruff
Jun 30 2017 18:12
Yeah, can't get past this exception in IL2CPP
Tried seeing if it was a timeout
logger - startup - timeout = 0:0:60
No dice
Just hangs indefinitely
Jesse Connor
@jesseconnr
Jun 30 2017 18:23
Did you build out a helper to feed the messages through another actor like they did with NLog? https://github.com/akkadotnet/Akka.Logger.NLog/blob/dev/src/Akka.Logger.NLog/NLogLogger.cs
Andy Kutruff
@akutruff
Jun 30 2017 18:24
```
 public class UnityLogger : ReceiveActor, IRequiresMessageQueue<ILoggerMessageQueueSemantics>
    {
        public static Action<string> LogAction;
        private readonly ILoggingAdapter _log = Context.GetLogger();

        public UnityLogger()
        {
            Receive<Error>(m => Handle(m));
            Receive<Warning>(m => Handle(m));
            Receive<Info>(m => Handle(m));
            Receive<Debug>(m => Handle(m));
            Receive<InitializeLogger>(m =>
            {
                _log.Info("UnityLogger started");
                Sender.Tell(new LoggerInitialized());
            });
        }

        private static string GetFormat(object message)
        {
            var logMessage = message as LogMessage;
            return logMessage != null ? logMessage.Format : "{Message}";
        }

        private static void Handle(Error logEvent)
        {
            LogIt(logEvent);
        }

        private static void Handle(Warning logEvent)
        {
            LogIt(logEvent);
        }

        private static void Handle(Info logEvent)
        {
            LogIt(logEvent);
        }

        private static void Handle(Debug logEvent)
        {
            LogIt(logEvent);

        }

        public static void LogIt(LogEvent logEvent)
        {
            if (logEvent == null || logEvent.Message == null)
                return;
            try
            {
                var logMessage = $"{logEvent.Message}";
                LogIt(logMessage);
            }
            catch
            {

            }
        }

        public static void LogIt(string logMessage)
        {
            if (LogAction != null)
            {
                LogAction(logMessage);
            }
        }
        public static void LogIt(object logMessage)
        {
            if (logMessage != null)
            {
                LogIt(logMessage.ToString());
            }
        }
    }
I kept it brain dead simple
It works in one environment but not the other
The static callback is the hack to just pump to unity's static method
I put in a lot of defensive things temporarily
It's just a matter of it actually getting created under IL2CPP. I'm not trusting their Task / async
Andy Kutruff
@akutruff
Jun 30 2017 18:54
Am I specifying the timeout correctly?
Jesse Connor
@jesseconnr
Jun 30 2017 19:08
how are you loading your config?
via app.config or parsing a string?
Andy Kutruff
@akutruff
Jun 30 2017 19:09
parsing string. I think it's in milliseconds actually
The config language
            var config = ConfigurationFactory.ParseString(@"
akka {  
    loglevel = DEBUG
    logger-startup-timeout = 1200000
    loggers = [""Chat.UnityLogger, Chat""]
}
");
^ for reference, I don't think they link to it anywhere that I've seen
Andy Kutruff
@akutruff
Jun 30 2017 19:10
Thanks, did come across that. Still couldn't hunt down this guy though
It appears to be affecting the error. So there is a deadlock during logger initialization somehow
Does Akka use a SynchronizationContext or custom TaskScheduler?
Or is it just using the default?
looks like you can specify
100s, 100ms, etc
Andy Kutruff
@akutruff
Jun 30 2017 19:20
Thanks, this is helpful.
Jesse Connor
@jesseconnr
Jun 30 2017 19:21
and yeah default is milliseconds
for your other question, I don't have any idea
might be able to play with the debug settings so it will break where the error is
Andy Kutruff
@akutruff
Jun 30 2017 19:24
I'm pretty sure it's hanging at that point. The error is delayed by that timer.
Jesse Connor
@jesseconnr
Jun 30 2017 19:30
Right, I mean having it break in library code, either using pdb or decompiling.
Andy Kutruff
@akutruff
Jun 30 2017 19:30
I'm dealing with IL that has been translated into C++
A bit of a nightmare.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 19:42
looking into the Akka.Remote bits now
have a bunch of specs I've spent the past couple of hours verifying different shutdown scenarios
so far it looks like Akka.Remote is doing the right thing
that being said, when there's smoke there's fire Azure/DotNetty#238
problem might be DotNetty itself
have one more scenario I need to test (failed outbound connection) and if that one passes then I'm going to have to brush the dust off of my fork of DotNetty
and see what's up there
Andy Kutruff
@akutruff
Jun 30 2017 19:44
Is DotNetty built on Sockets or TcpClient for tcp?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 19:44
SocketAsyncEventArgs
Andy Kutruff
@akutruff
Jun 30 2017 19:45
hmmmm
Aaron Stannard
@Aaronontheweb
Jun 30 2017 19:45
might want to have some of our users go back to using the Helios 2 transport if this is a persistent issue
which we ship as a separate nuget package now
was the default transport for Akka.NET for a long time
Andy Kutruff
@akutruff
Jun 30 2017 20:12
Well, I just searched for the word "lock" in dotnetty and didn't get a lot of results... :)
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:12
it uses the same type of CAS mechanisms that Akka does
Andy Kutruff
@akutruff
Jun 30 2017 20:12
ah, okay.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:12
the IChannel itself
is thread-safe and processes events in order
essentially it acts like an asynchronous event loop
so you don't need to synchronize any of the guts inside of that unless they can be concurrently modified from the outside
Andy Kutruff
@akutruff
Jun 30 2017 20:15
I just saw this as one of the latest commits and saw "concurrent" which is not my favorite sign
Azure/DotNetty@b345bb0
Andy Kutruff
@akutruff
Jun 30 2017 20:22
I'm compiling the project directly into my test project - do I just set a config to try helios?
and include an akka.remote.helios.tcp section that has the FQN of the Helios transport class listed
Andy Kutruff
@akutruff
Jun 30 2017 20:25
cool, thanks. Will be interesting to see if IL2CPP reacts differently
example of the helios FQN
the helios nuget package is installed separately
Andy Kutruff
@akutruff
Jun 30 2017 20:27
ReceiveActor... matchbuilder
Figured out how to redirect console.writeline directly without loggers
to see why my logger was failing
Found out there's a hidden AOT problem
You mentioned ReceiveActor yesterday
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:27
yeah, the matchBuilder in the receive actor
does a ton of stuff with expression compilation
Andy Kutruff
@akutruff
Jun 30 2017 20:28
[ERROR][6/30/2017 8:13:39 PM][Thread 0010][akka://ChatClient/system/log1-UnityLogger] Object reference not set to an instance of an object.
Cause: [akka://ChatClient/system/log1-UnityLogger#1518980873]: Akka.Actor.ActorInitializationException: Exception during creation ---> System.NullReferenceException: Object reference not set to an instance of an object.
  at System.Linq.Expressions.Expression.CreateLambda (System.Type delegateType, System.Linq.Expressions.Expression body, System.String name, System.Boolean tailCall, System.Collections.ObjectModel.ReadOnlyCollection`1[T] parameters) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Linq.Expressions.Expression.Lambda (System.Linq.Expressions.Expression body, System.String name, System.Boolean tailCall, System.Collections.Generic.IEnumerable`1[T] parameters) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Linq.Expressions.Expression.Lambda (System.Linq.Expressions.Expression body, System.Linq.Expressions.ParameterExpression[] parameters) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Tools.MatchHandler.MatchExpressionBuilder`1[T].BuildLambdaExpression (System.Collections.Generic.IReadOnlyList`1[T] typeHandlers) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Tools.MatchHandler.CachedMatchCompiler`1[T].CompileToDelegate (System.Collections.Generic.IReadOnlyList`1[T] handlers, System.Collections.Generic.IReadOnlyList`1[T] capturedArguments, System.Object[]& delegateArguments) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Tools.MatchHandler.CachedMatchCompiler`1+<>c__DisplayClass6_0[T].<Compile>b__0 (Akka.Tools.MatchHandler.MatchBuilderSignature _) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoProperty+GetterAdapter.Invoke (System.Object _this) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Collections.Concurrent.ConcurrentDictionary`2[TKey,TValue].GetOrAdd (TKey key, System.Func`2[T,TResult] valueFactory) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Tools.MatchHandler.CachedMatchCompiler`1[T].Compile (System.Collections.Generic.IReadOnlyList`1[T] handlers, System.Collections.Generic.IReadOnlyList`1[T] capturedArguments, Akka.Tools.MatchHandler.MatchBuilderSignature signature) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Tools.MatchHandler.MatchBuilder`1[TItem].Build () [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.ReceiveActor.Akka.Actor.Internal.IInitializableActor.Init () [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.ActorCell.CreateNewActorInstance () [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.ActorCell+<>c__DisplayClass115_0.<NewActor>b__0 () [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.GUISkin+SkinChangedDelegate.Invoke () [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.ActorCell.UseThreadContext (System.Action action) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.ActorCell.NewActor () [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.ActorCell.Create (System.Exception failure) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.ActorCell.SysMsgInvokeAll (Akka.Dispatch.SysMsg.EarliestFirstSystemMessageList messages, System.Int32 currentState) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Actor.ActorCell.SystemInvoke (Akka.Dispatch.SysMsg.ISystemMessage envelope) [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Dispatch.Mailbox.ProcessAllSystemMessages () [0x00000] in <00000000000000000000000000000000>:0 
  at Akka.Dispatch.Mailbox.<Run>b__36_0 () [0x00000] in <00000000000000000000000000000000>:0
That "UnityEngine." stuff in the callstack can be ignored. Their stack traces get confused some times
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:28
doesn't sound like an AOT issue
this time
Andy Kutruff
@akutruff
Jun 30 2017 20:29
CreateLambda
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:29
what's your actor definition look like?
Andy Kutruff
@akutruff
Jun 30 2017 20:30
   public class UnityLogger : ReceiveActor, IRequiresMessageQueue<ILoggerMessageQueueSemantics>
    {
        public static Action<string> LogAction;
        private readonly ILoggingAdapter _log = Context.GetLogger();

        public UnityLogger()
        {
            Receive<Error>(m => Handle(m));
            Receive<Warning>(m => Handle(m));
            Receive<Info>(m => Handle(m));
            Receive<Debug>(m => Handle(m));
            Receive<InitializeLogger>(m =>
            {
                _log.Info("UnityLogger started");
                Sender.Tell(new LoggerInitialized());
            });
        }

        private static string GetFormat(object message)
        {
            var logMessage = message as LogMessage;
            return logMessage != null ? logMessage.Format : "{Message}";
        }

        private static void Handle(Error logEvent)
        {
            LogIt(logEvent);
        }

        private static void Handle(Warning logEvent)
        {
            LogIt(logEvent);
        }

        private static void Handle(Info logEvent)
        {
            LogIt(logEvent);
        }

        private static void Handle(Debug logEvent)
        {
            LogIt(logEvent);

        }

        public static void LogIt(LogEvent logEvent)
        {
            if (logEvent == null || logEvent.Message == null)
                return;
            try
            {
                var logMessage = $"{logEvent.Message}";
                LogIt(logMessage);
            }
            catch
            {

            }
        }

        public static void LogIt(string logMessage)
        {
            if (LogAction != null)
            {
                LogAction(logMessage);
            }
        }
        public static void LogIt(object logMessage)
        {
            if (logMessage != null)
            {
                LogIt(logMessage.ToString());
            }
        }
    }
I've been removing all the expression props in the code base
as per yesterday
Well, changing to fluent form, not removing
This was a quick snap of the Serilog
So, it's a receive actor issue?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:33
might be... but Akka.Remote works on your app, right?
Andy Kutruff
@akutruff
Jun 30 2017 20:34
It works in the UnityEditor (not AOT) I have not gotten remote to work under IL2CPP (AOT)
When trying to get Remote to work, you suggested getting the logging, so I'm trying to put in a custom logger
And the custom logger is running into this issu
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:34
ah
because Akka.Remote uses a bunch of ReceiveActors internally
Andy Kutruff
@akutruff
Jun 30 2017 20:35
What's this matchbuilder stack about?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:35
first thing I'd try to rule out here is that there's something wrong with the way this actor is designed, although nothing jumps out at me
but the matchbuilder is the engine that the ReceiveActor uses
to compile all of its pattern-matching statements
does lambda caching and compilation
Andy Kutruff
@akutruff
Jun 30 2017 20:45
What are the lambdas extracting in the case?
Rather than just using straight up delegates?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:45
I didn't write any of that code
so it's a bit of a mystery to me
Andy Kutruff
@akutruff
Jun 30 2017 20:46
That's my favorite answer when I'm on the other end of questioning. :)
It's setting up a dispatch table for sure
Aaron Stannard
@Aaronontheweb
Jun 30 2017 20:50
@object so you're seeing issues with DotNetty sockets not closing correctly?
or at least not right away?
Andy Kutruff
@akutruff
Jun 30 2017 20:53
Did you get GC benchmarks?
Andy Kutruff
@akutruff
Jun 30 2017 21:02
        //Basically this class build this expression tree:
        //   bool Handler(T input, Type0 arg0, ..., Typen argn, object[] extra)
        //   {
        //      //BODY
        //      return: //label
        //   }
        //
        //We try to inline as many argument as function parameters as possible, the rest goes into the extra array.
        //The arguments are the predicates and handler methods captured by the builder.
        //   Example 1:
        //      Receive<string>(s=>s.Length==5, s=>DoSomething(s));
        //      Receive<string>(s=>DoSomethingElse(s));
        //      Receive<int>(i=>DoSomethingElseWithInt(i));
        //Will result in the following code:
        //   Example 2:
        //      bool Handler(T input, Predicate<string> p1, Action<string> a1, Action<string> a2, Action<int> a3)
        //      {
        //          var s = input as string;
        //          if(s != null)
        //          {
        //              if(p1(s))
        //              {
        //                  a1(s);
        //                  return true;
        //              }
        //              a2(s);
        //              return true;
        //          }
        //          if(input is int)
        //          {
        //              var i = (int)input;
        //              a3(i);
        //              return true;
        //          }
        //          return false;
        //      }
        //Typically the code in Example 1 is called in the constructor of an Actor, and although the code looks the same,
        //the actions and predicates sent in to receive actually are new instances.
        //The reason we capture the predicates and actions, instead of calling them directly, is that the code 
        //in Example 2, can be compiled, cached and reused no matter the predicates and actions.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:04
yeah I just gathered some from our dispatcher benchmark
Andy Kutruff
@akutruff
Jun 30 2017 21:04
Oh yeah, you sent some earlier. Sorry, long day
So this pattern matching stuff... It appears to be a way of avoiding a closure?
But it requires a stack regardless...
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:05

Akka.Tests.Performance.Dispatch.ThreadPoolDispatcherWarmThroughputSpec+Schedule_throughput

Tests how long it takes to schedule items onto the dispatcher
6/30/2017 6:55:27 PM

System Info

NBench=NBench, Version=1.0.1.0, Culture=neutral, PublicKeyToken=null
OS=Microsoft Windows NT 6.2.9200.0
ProcessorCount=2
CLR=4.0.30319.42000,IsMono=False,MaxGcGeneration=2

NBench Settings

RunMode=Iterations, TestMode=Measurement
NumberOfIterations=5, MaximumRunTime=00:00:01
Concurrent=True
Tracing=True

Data


Totals

Metric Units Max Average Min StdDev
TotalCollections [Gen0] collections 1,680.00 1,666.80 1,661.00 7.82
TotalCollections [Gen1] collections 628.00 571.60 551.00 32.01
TotalCollections [Gen2] collections 68.00 29.20 11.00 22.82
[Counter] ScheduledActionCompleted operations 100,000,000.00 100,000,000.00 100,000,000.00 0.00

Per-second Totals

Metric Units / s Max / s Average / s Min / s StdDev / s
TotalCollections [Gen0] collections 81.68 72.13 63.23 8.55
TotalCollections [Gen1] collections 27.24 24.71 20.98 2.90
TotalCollections [Gen2] collections 2.81 1.27 0.43 0.95
[Counter] ScheduledActionCompleted operations 4,905,855.77 4,327,200.38 3,806,721.51 510,945.98

Raw Data

TotalCollections [Gen0]

Run # collections collections / s ns / collections
1 1,665.00 81.68 12,242,524.62
2 1,661.00 65.67 15,227,933.96
3 1,680.00 69.42 14,405,695.65
4 1,667.00 80.65 12,399,617.10
5 1,661.00 63.23 15,815,366.53

TotalCollections [Gen1]

Run # collections collections / s ns / collections
1 553.00 27.13 36,860,404.16
2 563.00 22.26 44,926,462.34
3 628.00 25.95 38,537,529.78
4 563.00 27.24 36,714,319.18
5 551.00 20.98 47,675,723.77

TotalCollections [Gen2]

Run # collections collections / s ns / collections
1 26.00 1.28 783,992,442.31
2 11.00 0.43 2,299,418,027.27
3 68.00 2.81 355,905,422.06
4 27.00 1.31 765,561,544.44
5 14.00 0.53 1,876,380,271.43

[Counter] ScheduledActionCompleted

Run # operations operations / s ns / operations
1 100,000,000.00 4,905,855.77 203.84
2 100,000,000.00 3,953,569.55 252.94
3 100,000,000.00 4,131,963.56 242.02
4 100,000,000.00
Andy Kutruff
@akutruff
Jun 30 2017 21:06
It came with the first commit of ReceiveActor, so I can't see what problem it was trying to solve.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:06
problem we were trying to solve was we didn't have a good way of doing pattern matching four years ago
the PatternMatch class allocated per-message
and the if msg is Foo.... var f = (Foo)msg stuff was hideous to write and maintain
for applications that had to handle more than a few message types
Andy Kutruff
@akutruff
Jun 30 2017 21:07
For the callback table though, it's just Dictionary<Type, Thing>
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:07
IMHO
Andy Kutruff
@akutruff
Jun 30 2017 21:07
To achieve the same API for Receive
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:07
totally possible that thing was over-engineered
knowing the guy who wrote it
Andy Kutruff
@akutruff
Jun 30 2017 21:08
hehe, I was trying to be exploratory and tactful before being declarative, but that was my instant first impression
It actually looks more expensive at the end of the day
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:08
ironically enough
the benchmarks show the actor performance to be comparable
but
I just realized
that's probably because the dispatcher is the bottle neck
not the actor class implementation itself
Andy Kutruff
@akutruff
Jun 30 2017 21:10
I'd really set a project goal to bring gen 2 and gen 1 down to zero
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:10
some of the allocations on the dispatcher are unavoidable
queueing something onto the ThreadPool will do it
but I suspect there's things we can do to improve the state of affairs there
Andy Kutruff
@akutruff
Jun 30 2017 21:11
Is the threadpool because of .NET itself? Like io completion?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:11
yeah
it's built into the CLR
Task et al use it to run
and there are the IOCP threads as well
but they're segregated from the rest of the thread pool
Andy Kutruff
@akutruff
Jun 30 2017 21:12
Oh, well, Task you can make a TaskScheduler
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:12
since it'd be.... bad... if people could schedule blocking work onto them
Andy Kutruff
@akutruff
Jun 30 2017 21:13
Are you able to see how much context switching is going on?
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:13
haven't measured that much - when an actor processes a batch of messages there's no context switching at all there
since it happens on the same thread
Andy Kutruff
@akutruff
Jun 30 2017 21:13
The TaskScheduler I use for a actors reserves #Cores threads. I find the ThreadPool ends up thrashing
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:14
but when it's different actors being interleaved
then yeah, there's going to be some context switching
Andy Kutruff
@akutruff
Jun 30 2017 21:14
I will use a threadpool thread for kicking off a root timer though. I found it to be the most accurate.
The code doesn't appear to be too task heavy though
So the Task (as a class) overhead, I'd assume would tend to stay gen 0
Alright, so I'm going to change ReceiveActor to use a plain old dispatch table to see if that unblocks me.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:17
we don't use the TPL inside the dispatcher at all
Andy Kutruff
@akutruff
Jun 30 2017 21:17
oh, word
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:18
it's just overhead for that stuff
Andy Kutruff
@akutruff
Jun 30 2017 21:19
Sorry if I'm consuming time and dumping here too much, btw. Really want to use this lib, and have a big checklist from having our own hand rolled.
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:19
all good man
we want to support Unity
and if there's a road to getting there that ends up improving everyone else's performance in the process
win-win
_shouldUnhandle
Lealand Vettleson
@spankr
Jun 30 2017 21:45
I noticed a weird log message in our tests. When testing our Tcp connection client actor (based on http://getakka.net/docs/IO), we see a large number of unloaded AppDomain messages. (using xunit) If I give the actor an extra second to wait in the PostStop, all but one of those messages clean up.
I assume there must be some underlying work going on when cleaning up the tcp stuff
Aaron Stannard
@Aaronontheweb
Jun 30 2017 21:56
hmmm
by default XUnit will use AppDomains for test isolation
could be that stuff is still running in the background when it gets unloaded
or a socket waiting on a Select call
which is how Akka.IO is implemented currently
Andy Kutruff
@akutruff
Jun 30 2017 23:06
baby steps. Rewrote that stuff. ReceiveActor now not barfing during AOT. DotNetty now having endpoint complaints under AOT.
Will switch to helios before trying to see what's up with DotNetty