These are chat archives for akkadotnet/akka.net

19th
Nov 2015
Bartosz Sypytkowski
@Horusiath
Nov 19 2015 07:58
@cduhard Once we fix issues with our build server, could you try out your issue on the latest version of Akka.Persistence? There were a lot of bug fixes in this area since 1.0.4
Roger Johansson
@rogeralsing
Nov 19 2015 08:12
@Horusiath I've published a new version of wire on nuget with our latest changes
chillitom
@chillitom
Nov 19 2015 09:10
with persistent views, is there a way to wildcard the persistent id/name? I'd like to write a projection across many actors' event streams?
Roger Johansson
@rogeralsing
Nov 19 2015 09:37
@chillitom afaik it is not possible, persistent views are 1-1 , thats why Typesafe is redesigning this using Akka Streams on the JVM to enable views that are aggregations of multiple sources
but @Horusiath would know more
Samuel Verstraete
@elangelo
Nov 19 2015 09:46
anyone ever tried to host WF workflows in akka?
or is there a WF alike functionality?
i don't need the visual stuff of it
just the persistence and the modularity
chillitom
@chillitom
Nov 19 2015 09:56
@rogeralsing thanks, that's interesting, be great to see similar in akka.net ... happy to help out
Q: what's the correct way to configure custom serialization for Akka.Persistence? What interfaces should I implement?
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 10:08
@Aaronontheweb So from the logs I can tell that the process for the test runner completes.
process.WaitForExit() returns
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 10:52
If an actor is setup with Receive<T>( x => DoSomethingAsync(x)); am I right in thinking that DoSomethingAsync is not going to be observed?
it'll just be a fire and forget into the ether.
Bartosz Sypytkowski
@Horusiath
Nov 19 2015 11:53
@chillitom every persistent journal backend may define it's own set of queries, you may use to get overall information across multiple persistent ids
this is in fact what sql-based journals will support soon
Bartosz Sypytkowski
@Horusiath
Nov 19 2015 12:01
and when it comes to serializers:
  1. You need to create your own serializer inherinting from Akka.Serialization.Serializer it has to have unique Id in scope of an actor system
  2. You'll need to register it in HOCON config using i.e. akka.actor.serializers.my-serializer-alias = "MyNamespace.MySerializerClass, MyAssembly"
  3. Bind it to particular type of message in HOCON using akka.actor.serialization-bindings { "MyNamespace.MyClass, MyAssembly" = my-serializer-alias }. This binding is inheritance-sensitive and you may apply it to interface, that you can later use to mark each particular data type to be serialized/deserialized with your custom serializer
chillitom
@chillitom
Nov 19 2015 12:45
@Horusiath would there be a way I could bind it to all F# discriminated unions?
Bartosz Sypytkowski
@Horusiath
Nov 19 2015 13:24
unfortunatelly not, as discriminated unions don't have single common interface or root. But i.e. you could attach an interface like that:
type MyUnion =
    | A
    | B
    interface ISerializeWithFlatBuffers
Thomas Lazar
@thomaslazar
Nov 19 2015 14:26
trying to get a test to log stuff out with nlog so i can see what stuff is happening. but i can't seem to get it to work. did i do anything wrong there?
        public SearchIndexMasterTest()
            : base(@"
                    akka.loglevel = INFO
                    loggers = [""Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog""]
        ")
        {
        }
i have a nlog.config and stuff. and on my web project it's working. just not on the test.
nevermind... i was stoopid
forgot the "akka." before "loggers"
Christian Duhard
@cduhard
Nov 19 2015 15:13
@Horusiath I will try it as soon as a new release is up
@Hori
@Horusiath it's definitely broken right now. When a lot of messages are processed quickly the state doesn't get updated until a pause in messages throughput
Christian Duhard
@cduhard
Nov 19 2015 15:20
Have there been any memory leak issues?
Thomas Lazar
@thomaslazar
Nov 19 2015 15:46
can i specify more then one logger in akka? when i change the logger in my tests to nlog it removes the stdout logger and the EventFilter thing seems to break
Christian Duhard
@cduhard
Nov 19 2015 16:02
does the Scheduler leak memory by chance?
Christian Duhard
@cduhard
Nov 19 2015 16:09
ScheduledWork is steadily consuming memory
Christian Duhard
@cduhard
Nov 19 2015 16:19
my service runs out of memory just sitting idle running a few repeating schedules in parallel
Aaron Stannard
@Aaronontheweb
Nov 19 2015 16:32
@GraemeBradbury if the lambda in the Receive<T>(async x => DoSomethingAsync(x)) then the actor will use its async await
looks like all of the specs that have had high failure / repro rates are ones that use Ask or TPL
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 16:35
@Aaronontheweb That'll halt processing of messages until the await returns?
Aaron Stannard
@Aaronontheweb
Nov 19 2015 16:35
correct
all of this started happening after I upgraded the build server - doesn't look like it was any of the PRs.... I wonder if RyuJIT is running under the covers
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 16:36
You upgraded to VS2015 right?
Aaron Stannard
@Aaronontheweb
Nov 19 2015 16:36
yep
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 16:38
I'm pretty sure that'll turn ryujit on.
Aaron Stannard
@Aaronontheweb
Nov 19 2015 16:38
works locally for me though, using MSBuild from 2015
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 16:40
Same here. I get no test failures but push it to the build server and I get unit tests failing.
Driving me up the wall.
Aaron Stannard
@Aaronontheweb
Nov 19 2015 16:40
tell you what
I'll fire up a build agent manually and give you some RDP credentials
you should be able to do stuff like remote debugging on it
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 16:41
no rush, won't be able to do much till kids are asleep in ~3hours.
Aaron Stannard
@Aaronontheweb
Nov 19 2015 16:43
np
firing one up now
going to intentionally not let it connect to TeamCity
but you can still go into it and git clone + build stuff
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 16:44
thanks
Aaron Stannard
@Aaronontheweb
Nov 19 2015 16:45
it's running the exact same configuration as the agents
although it might be a little slow for you to connect since the server is in Texas :p
chillitom
@chillitom
Nov 19 2015 16:52
@cduhard can you give some more information about what you're trying to do?
Christian Duhard
@cduhard
Nov 19 2015 16:59
I am using akka persistence
i have a few repeating schedules running and memory usgae goes nuts after running for a while
the only messages are the scheduled ones
Sytem.Collections.Generic.List<Akka.Actor.SheduledWork> seems to be the culprit
chillitom
@chillitom
Nov 19 2015 17:19
@cduhard are you scheduling too much work? maybe you're scheduling a repeated message when you mean to schedule just one?
Christian Duhard
@cduhard
Nov 19 2015 17:20
nope, it's repeated
chillitom
@chillitom
Nov 19 2015 17:20
without some code we can only guess
Christian Duhard
@cduhard
Nov 19 2015 17:21
i am going to profile a bit more, if i can't figure it out i'll provide some more info
Aaron Stannard
@Aaronontheweb
Nov 19 2015 18:11
@GraemeBradbury the speed at which the test suite gets executed on our build agents is appallingly slow
what CPUs are they running these things on? Potatoes?
Christian Duhard
@cduhard
Nov 19 2015 18:26
turnips
Christian Duhard
@cduhard
Nov 19 2015 18:38
It seems there was a weird race between 2 scheduled messages within an actor causing the memory explosion
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:05
BenchmarkDotNet=v0.7.8.0
OS=Microsoft Windows NT 6.2.9200.0
Processor=Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz, ProcessorCount=2
HostCLR=MS.NET 4.0.30319.42000, Arch=32-bit 
Type=Sdk_ReflectionBenchmarkInvoker  Mode=Throughput  .NET=HostFramework
              Method | Platform |       Jit |    AvrTime |    StdDev |          op/s |
------------------------ |--------- |---------- |----------- |---------- |-------------- |
InvokeRunWithContext | X64 | LegacyJit | 10.5523 ns | 1.1035 ns | 94,766,468.79 |
InvokeRunWithoutContext | X64 | LegacyJit | 14.6422 ns | 1.1389 ns | 68,295,720.50 |
InvokeRunWithContext | X64 | RyuJit | 10.6522 ns | 1.0664 ns | 93,877,314.01 |
InvokeRunWithoutContext | X64 | RyuJit | 14.2104 ns | 1.3392 ns | 70,374,765.92 |
InvokeRunWithContext | X86 | LegacyJit | 13.3760 ns | 1.1300 ns | 74,767,355.46 |
InvokeRunWithoutContext | X86 | LegacyJit | 22.0173 ns | 1.8681 ns | 45,418,865.71 |
build server data
my local machine
BenchmarkDotNet=v0.7.8.0
OS=Microsoft Windows NT 6.2.9200.0
Processor=Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz, ProcessorCount=8
HostCLR=MS.NET 4.0.30319.42000, Arch=32-bit  [AttachedDebugger]
Type=Sdk_ReflectionBenchmarkInvoker  Mode=Throughput  .NET=HostFramework
              Method | Platform |       Jit |   AvrTime |    StdDev |           op/s |
------------------------ |--------- |---------- |---------- |---------- |--------------- |
InvokeRunWithContext | X64 | LegacyJit | 2.9036 ns | 0.0636 ns | 344,396,149.89 |
InvokeRunWithoutContext | X64 | LegacyJit | 4.1686 ns | 0.0827 ns | 239,889,965.00 |
InvokeRunWithContext | X64 | RyuJit | 2.9046 ns | 0.0319 ns | 344,278,130.72 |
InvokeRunWithoutContext | X64 | RyuJit | 4.1809 ns | 0.1283 ns | 239,185,464.46 |
InvokeRunWithContext | X86 | LegacyJit | 3.8308 ns | 0.0928 ns | 261,042,845.95 |
InvokeRunWithoutContext | X86 | LegacyJit | 5.7497 ns | 0.0489 ns | 173,922,434.40 |
these are single threaded specs
from a separate project
the build agent and my dev machine have the same clock speed
2.40GHz
but across the board my machine outperforms the VM on throughput by 4-5x
I'm using beefy i7s locally
Trying to sort out what can account for the difference
Sean Gilliam
@sean-gilliam
Nov 19 2015 19:20
sorry I had my bitcoin miners running on them :P
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:27
might be getting closer to the heart of the issue
looks like Azure has been underreporting CPU utilization on these machines
all of the DI tests run with like 99% CPU utilization
and since I use the TPL in those specs
there's a chance the return Task never gets sent on time under those circumstances
Stefan Sedich
@stefansedich
Nov 19 2015 19:29
wowsers nice
time for some supercomputers
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:29
meanwhile most of the Akka.Persistence specs and Akka.Remote ones run with like 10% CPU utilization
not all of them, there are some that spike up into the 90s
Stefan Sedich
@stefansedich
Nov 19 2015 19:29
oh DI tests too
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:29
but if writing to the network is cheaper than using your DI framework
Stefan Sedich
@stefansedich
Nov 19 2015 19:29
interesting
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:30
seriously, abandon that shit
take the DI framework out back and shoot it in the head
Stefan Sedich
@stefansedich
Nov 19 2015 19:30
lol
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:30
looking at you, StructureMap
number one offender
Stefan Sedich
@stefansedich
Nov 19 2015 19:30
wtf are they doing to spike cpu so hard
I bet Autofac doesn't give us grief right :D
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:31
could also be XUnit
I've literally never seen an Autofac spec fail
not since I fixed a core bug inside DI
as part of that PR
Stefan Sedich
@stefansedich
Nov 19 2015 19:31
yeah it "just works"
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:31
Unity, StructureMap though
mad sketch
oh shit lol
got the multi-node test runner to run on one of the boxes
pegged at 100%
the little build server just can't hang with that
here's the part that drives me nuts - these tests don't do all that much
unless the actual guts of the TestKit have a shitton of GC overhead or something
or the Akka.Remote.TestKit
one of the things I did this week was shrink the VM size way down
we used to run each agent with 8 cores or something like that
no idea what I was smoking when I set that up
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:36
dropped it down to 2
welp, was able to recreate the deadlock in the multi-node test runner
looks like it's an individual Akka.NodeTestRunner.exe instance
holding steady at ~70% CPU utilization
for a couple of minutes now
time to turn on remote VS debugging I think
Aaron Stannard
@Aaronontheweb
Nov 19 2015 19:49
firing up a third VM - same core count, but the fastest CPUs I can get in the Southwest Azure data center
want to see how much of this issue comes down to CPU utilization %
Kamil Wojciechowski
@aph5nt
Nov 19 2015 20:14
how is it going with akka.persistance ?
Stefan Sedich
@stefansedich
Nov 19 2015 20:40
are those boxes a "Shared cpu" ?
Aaron Stannard
@Aaronontheweb
Nov 19 2015 20:48
BenchmarkDotNet=v0.7.8.0
OS=Microsoft Windows NT 6.2.9200.0
Processor=Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz, ProcessorCount=2
HostCLR=MS.NET 4.0.30319.42000, Arch=32-bit
Type=Sdk_ReflectionBenchmarkInvoker  Mode=Throughput  .NET=HostFramework
              Method | Platform |       Jit |   AvrTime |    StdDev |           op/s |
------------------------ |--------- |---------- |---------- |---------- |--------------- |
InvokeRunWithContext | X64 | LegacyJit | 4.1302 ns | 0.2116 ns | 242,118,202.37 |
InvokeRunWithoutContext | X64 | LegacyJit | 6.1375 ns | 0.3170 ns | 162,932,191.53 |
InvokeRunWithContext | X64 | RyuJit | 4.2583 ns | 0.3134 ns | 234,836,831.78 |
InvokeRunWithoutContext | X64 | RyuJit | 6.0838 ns | 0.3993 ns | 164,380,255.89 |
InvokeRunWithContext | X86 | LegacyJit | 5.5112 ns | 0.3445 ns | 181,450,184.52 |
InvokeRunWithoutContext | X86 | LegacyJit | 8.4663 ns | 0.3887 ns | 118,116,850.55 |
these are the high end Azure machines
the earlier one was from the shitty basic one
the "high end" CPU has much higher cache values
than this one
which is what the other VM was using
but between you and me @stefansedich - those cache differences alone can't explain the dramatic difference in benchmark values
I suspect those basic machines must have shared cores
Stefan Sedich
@stefansedich
Nov 19 2015 20:51
yeah
that was my assumption
but we know how dangerous those can be :D
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 20:55
How is the processor count calculated? surely they don't just give a single core for that Xeon?
Aaron Stannard
@Aaronontheweb
Nov 19 2015 20:55
let me check the source
but yeah, Azure said "2 cores"
on the website
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 20:57
since the i7 said 8 i'm guessing ProcessorCount is virtual cores in which case that Xeon has 8 vm's assigned to it, each on a separate core.
Aaron Stannard
@Aaronontheweb
Nov 19 2015 20:58
Environment.ProcessorCount
which I believe is the number of "logical" processors
those Xeons have hyper-threading, right?
Stefan Sedich
@stefansedich
Nov 19 2015 20:59
think so
the number is "logical cores"
Aaron Stannard
@Aaronontheweb
Nov 19 2015 20:59
so in reality
we might only be using one physical core
of that Xeon
Graeme Bradbury
@GraemeBradbury
Nov 19 2015 20:59
yup 8 physical cores, 16 "logical"
Aaron Stannard
@Aaronontheweb
Nov 19 2015 20:59
.... sigh
so in the event of that basic sever
Stefan Sedich
@stefansedich
Nov 19 2015 21:00
correct @Aaronontheweb just put in you ccard number and dial it up to 11
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:00
we're probably not even getting 100% reserved access to that 1 core
Stefan Sedich
@stefansedich
Nov 19 2015 21:00
yup
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:00
what a pile of shit
Stefan Sedich
@stefansedich
Nov 19 2015 21:00
"yay cloud"
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:00
"JUST CLOUD IT"
kind of hard to guarantee that a specific piece of code in a unit test can execute in < 3s with a noisy neighbor problem potentially
might run that benchmark again on an AWS box just to see if there's a big difference in hardware allocation there
probably isn't
Stefan Sedich
@stefansedich
Nov 19 2015 21:02
the cheap AWS will have the same issue
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:02
at least AWS is upfront about whether or not the cores get shared or not
Stefan Sedich
@stefansedich
Nov 19 2015 21:02
yeah I went looking for that info on azure
suprised
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:02
in the event of the super cheap stuff
Stefan Sedich
@stefansedich
Nov 19 2015 21:03
I would try a C4.large
or maybe an xlarge
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:03
oh Azure, you cray
well, the point is a bit moot anyway - was still able to reproduce the deadlock on the higher end Azure machine
but at least I know why our tests appear to be so "racy" on the build server
running soft real-time requirement code on shared cores is a risky enterprise
Stefan Sedich
@stefansedich
Nov 19 2015 21:04
there is a startup idea
"we are disrupting the cloud by offering real-time reliability on shared cores in the enteprise"
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:06
we are disrupting the cloud by offering real, actual hardware that you exclusively use
disrupting the future with the past
Stefan Sedich
@stefansedich
Nov 19 2015 21:07
woah, 1999 called and they want their rack back.
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:08
Pepperidge farm remembers
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:20
@GraemeBradbury I think this might be the problem
foreach (var process in processes)
            {
                process.WaitForExit();
                var exitCode = process.ExitCode;
                process.Close();
            }
in all cases thus far where I've been able to replicate the deadlock on one of these servers
the NodeTestRunner.exe hangs around
so I'm thinking it could be an issue with the individual NodeTestRunner.exe as well as this
doesn't look like the code that's blocking is the actual "write stuff to disk" code
let me see if I can timeout the process on WaitForExit
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:30
yeah, I think between that and waiting on the Sink in the NodeTestRunner
there's some condition where the Sink never gets the ITestAssemblyFinished message
so that process deadlocks
and then the multi-node test runner deadlocks when it attempts to reap all of the child processes
stevemesser
@stevemesser
Nov 19 2015 21:45

I am finally trying to apply a router configuration to my POC.
I can't tell from the log output one way or the other if it is configured correctly.
I can even put the name of a non existent actor and I still don't get any indication of such.
Should I see some indication of success/failure when configuring a router?

Here is my config:

<![CDATA[
akka
{
loggers = ["Akka.Logger.slf4net.Slf4NetLogger,Akka.Logger.slf4net"]

  stdout-loglevel = DEBUG
  loglevel = DEBUG
  log-config-on-start = on
  actor
  {
    debug
    {
      receive = on
      autorecieve = on
      lifecycle = on
      event-stream = on
    }
  }

  /datastore
  {
      router = round-robin-pool
      nr-of-instances = 5
  }
}

]]>

stevemesser
@stevemesser
Nov 19 2015 21:52
Should the datastore's Prestart method get called once for each instance in the size of the pool?
Aaron Stannard
@Aaronontheweb
Nov 19 2015 21:57
Running 393297 version of .NET 4.6 on build servers
Mikey
@BrainSlugs83
Nov 19 2015 22:52
Hey guys -- simple question with routers -- I have routers that I'm configuring with .WithRouter(FromConfig.Instance) -- and those actors have their routers as well -- also configured with .WithRouter(FromConfig.Instance) -- the first router is easy to configure in the HOCON because the path is predictable -- but how to do I configure the nested actors routers with HOCON?
Aaron Stannard
@Aaronontheweb
Nov 19 2015 22:58
@BrainSlugs83 wild card actor selections is the way to go
which you can specify in HOCON
an example
Mikey
@BrainSlugs83
Nov 19 2015 22:59
Neat, I was unaware of that -- is it just asterisk? -- like /Router1/*/Router2?
Aaron Stannard
@Aaronontheweb
Nov 19 2015 22:59
yep
Mikey
@BrainSlugs83
Nov 19 2015 22:59
POIFECT! :D
Aaron Stannard
@Aaronontheweb
Nov 19 2015 23:00
@GraemeBradbury dev branch is passing again since rolling back
Christian Duhard
@cduhard
Nov 19 2015 23:02
How do you do wild card in code?
Aaron Stannard
@Aaronontheweb
Nov 19 2015 23:02
just as a string I guess
when you create the ActorSelection
oh, I see what you mean
during deployments?
or just using ActorSelectionin general?
Christian Duhard
@cduhard
Nov 19 2015 23:04
if i am declaring a multiple router pools
say i have 3 children with that are provided their own pool of workers
*-with
Aaron Stannard
@Aaronontheweb
Nov 19 2015 23:06
@stefansedich rewriting all of our DI specs to use blocking methods instead of async / await
to see if that solves some of our issues with them being flakey
Stefan Sedich
@stefansedich
Nov 19 2015 23:06
isn't async faster @Aaronontheweb ?
Aaron Stannard
@Aaronontheweb
Nov 19 2015 23:08
I'm increasingly suspicious of it in high contention environments
Stefan Sedich
@stefansedich
Nov 19 2015 23:08
I await your verdict
Aaron Stannard
@Aaronontheweb
Nov 19 2015 23:08
times out
Aaron Stannard
@Aaronontheweb
Nov 19 2015 23:14
I've logged 11 new bugs in the past 24 hours
good lord
most of them look like issues with the way unit tests are written