Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 21:44
    Aaronontheweb commented #4054
  • 21:44
    Aaronontheweb commented #4054
  • 21:33
    MaximG1234 commented #4054
  • 21:31
    Aaronontheweb commented #4054
  • 21:31
    MaximG1234 commented #4054
  • 21:31
    MaximG1234 commented #4054
  • 20:04
    Aaronontheweb synchronize #4022
  • 20:03

    Aaronontheweb on dev

    NRE fix (#4056) (compare)

  • 20:03
    Aaronontheweb closed #4056
  • 20:03
    Aaronontheweb closed #4035
  • 19:53
    IgorFedchenko commented #4056
  • 19:44
    IgorFedchenko opened #4056
  • 18:50
    Aaronontheweb commented #4045
  • 18:42
    djm132 commented #4045
  • 17:49
    Aaronontheweb labeled #4044
  • 17:49
    Aaronontheweb labeled #4044
  • 17:49
    Aaronontheweb commented #4044
  • 17:49
    Aaronontheweb commented #4045
  • 17:43
    Aaronontheweb closed #4053
  • 17:43
    Aaronontheweb commented #4053
Bartosz Sypytkowski
@Horusiath
yes
jalchr
@jalchr
very well...
In all cases, how can we query a shard for its 'state' :) ?
jalchr
@jalchr
I'm experience "shortage" in communication ....
here is a sample log file
2017-12-09 16:03:35,998 [6] WARN  Akka.Cluster.ClusterCoreDaemon - Cluster Node [akka.tcp://ArchiveSystem@140.125.4.1:16568] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://ArchiveSystem@140.125.4.2:16668, Uid=81451205 status = Up, role=[importer-old], upNumber=21), Member(address = akka.tcp://ArchiveSystem@140.125.4.3:16666, Uid=807193699 status = Up, role=[web], upNumber=19)]. Node roles [importer,pubsub-node]
2017-12-09 16:03:36,013 [6] INFO  Akka.Cluster.ClusterCoreDaemon - Marking node(s) as REACHABLE [Member(address = akka.tcp://ArchiveSystem@140.125.4.2:16668, Uid=81451205 status = Up, role=[importer-old], upNumber=21)]. Node roles [importer,pubsub-node]
2017-12-09 16:03:36,013 [6] INFO  Archive.Shared.Cluster.ClusterStatus - UnreachableMember: Member(address = akka.tcp://ArchiveSystem@140.125.4.2:16668, Uid=81451205 status = Up, role=[importer-old], upNumber=21), Role(s): importer-old
2017-12-09 16:03:36,013 [6] INFO  Archive.Shared.Cluster.ClusterStatus - UnreachableMember: Member(address = akka.tcp://ArchiveSystem@140.125.4.3:16666, Uid=807193699 status = Up, role=[web], upNumber=19), Role(s): web
2017-12-09 16:03:36,013 [6] INFO  Akka.Cluster.ClusterCoreDaemon - Leader can currently not perform its duties, reachability status: [Reachability([akka.tcp://ArchiveSystem@140.125.4.1:16568 -> UniqueAddress: (akka.tcp://ArchiveSystem@140.125.4.2:16668, 81451205): Reachable [Reachable] (29)], [akka.tcp://ArchiveSystem@140.125.4.1:16568 -> UniqueAddress: (akka.tcp://ArchiveSystem@140.125.4.3:16666, 807193699): Unreachable [Unreachable] (28)])], member status: [$akka.tcp://ArchiveSystem@140.125.4.1:16568 $Up seen=$True, $akka.tcp://ArchiveSystem@140.125.4.1:16668 $Up seen=$False, $akka.tcp://ArchiveSystem@140.125.4.2:16568 $Up seen=$False, $akka.tcp://ArchiveSystem@140.125.4.2:16668 $Up seen=$False, $akka.tcp://ArchiveSystem@140.125.4.3:16666 $Up seen=$False, $akka.tcp://ArchiveSystem@140.125.4.4:16666 $Up seen=$False, $akka.tcp://ArchiveSystem@140.125.4.5:4053 $Up seen=$False, $akka.tcp://ArchiveSystem@140.125.4.6:4053 $Up seen=$False]
2017-12-09 16:03:36,138 [6] INFO  Archive.Shared.Cluster.ClusterStatus - ReachableMember: Member(address = akka.tcp://ArchiveSystem@140.125.4.2:16668, Uid=81451205 status = Up, role=[importer-old], upNumber=21), Role(s): importer-old
2017-12-09 16:03:36,138 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address: 
2017-12-09 16:03:36,138 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address: akka.tcp://ArchiveSystem@140.125.4.3:16666, 
2017-12-09 16:03:36,138 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address: akka.tcp://ArchiveSystem@140.125.4.3:16666, 
2017-12-09 16:03:36,154 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address: akka.tcp://ArchiveSystem@140.125.4.3:16666, 
2017-12-09 16:03:36,154 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address: akka.tcp://ArchiveSystem@140.125.4.3:16666, 
2017-12-09 16:03:36,154 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address: akka.tcp://ArchiveSystem@140.125.4.3:16666, 
2017-12-09 16:03:36,154 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address: akka.tcp://ArchiveSystem@140.125.4.3:16666, 
2017-12-09 16:03:36,154 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address: akka.tcp://ArchiveSystem@140.125.4.3:16666, 
2017-12-09 16:03:36,154 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address: akka.tcp://ArchiveSystem@140.125.4.3:16666, 
2017-12-09 16:03:36,170 [6] WARN  Archive.Shared.Cluster.ClusterStatus - Unreachable Member; Role: web, Status: Up, Address:
I have more than 2 roles ... "web" which is the front-end web application and "importer" which is the video processing service.
I noticed that after few days of running ... the combination of lighthouse and other roles ... somehow cause this
Bartosz Sypytkowski
@Horusiath
where are you hosting it?
jalchr
@jalchr
This causes even internal (no remoting) message forwarding to not reach its destination
I'm hosting this on 6 VMs (2 web , 2 processing ... 2 lighthouse)
and for the record, the IIS recycles the web app after some idle time ... so it is natural to become 'unreachable'
I wish I can reproduce this
Bartosz Sypytkowski
@Horusiath
maybe try to add IIS hook to allow node to gracefully leave the cluster before going down?
unreachable is normal part when machine crashes or network connection goes down - if its possible its better to gracefully signal if node is going offline.
jalchr
@jalchr
This is what I do in IIS ... asp.net app
//==============================================================
            HandleStarted();

            var properties = new AppProperties(app.Properties);
            var token = properties.OnAppDisposing;

            if (token != CancellationToken.None)
            {
                token.Register(() =>
                {
                    // do stuff here for ending / disposing
                    HandleStopped();
                    log.Info("ASP.NET application stopped !");
                });
            }

            log.Info("ASP.NET application started !");
Bartosz Sypytkowski
@Horusiath
also there are several cases for non-graceful scenarios
jalchr
@jalchr
private void HandleStopped()
        {
            log.Info("ASP.NET application stopped");
            if (ClusterSystem == null) return;
            log.Debug("Leaving cluster");
            ClusterHelper.Tell(new ClusterHelper.RemoveMember());
            var cluster = Akka.Cluster.Cluster.Get(ClusterSystem);
            cluster.RegisterOnMemberRemoved(() => MemberRemoved(ClusterSystem));
            cluster.Leave(cluster.SelfAddress);

            asTerminatedEvent.WaitOne(20000);
            ClusterSystem.Terminate().Wait(20000);
            log.Info("Actor system terminated, exiting");
        }

        private async void MemberRemoved(ActorSystem actorSystem)
        {
            log.Info("Member removed ...");
            await actorSystem.Terminate();
            this.asTerminatedEvent.Set();
        }
Internal messages fail:
2017-12-09 16:03:43,795 [56] INFO  Akka.Actor.LocalActorRef - Message Shutdown from akka://ArchiveSystem/system/sharding/FileHandler/67/%5C%5Cnas20%5Cd%24%5Carchive%5Cincoming%5Creuters%5C201712096110wd-northkorea-missilescontributors.xml/$c to akka://ArchiveSystem/system/sharding/FileHandler/67/%5C%5Cnas20%5Cd%24%5Carchive%5Cincoming%5Creuters%5C201712096110wd-northkorea-missilescontributors.xml/$c was not delivered. 4 dead letters encountered.
2017-12-09 16:03:43,826 [56] INFO  Akka.Actor.LocalActorRef - Message FileValidationResult from akka://ArchiveSystem/system/sharding/FileHandler/67/%5C%5Cnas20%5Cd%24%5Carchive%5Cincoming%5Creuters%5C201712096110wd-northkorea-missilescontributors.xml/$c to akka://ArchiveSystem/system/sharding/FileHandler/67/%5C%5Cnas20%5Cd%24%5Carchive%5Cincoming%5Creuters%5C201712096110wd-northkorea-missilescontributors.xml was not delivered. 5 dead letters encountered.
jalchr
@jalchr
Some additional notes, my applications are CPU and Network intensive apps. It is typical to have 100% CPU and 100% Network utilization, most of the time.
Perhaps this effects how akka.net communication works and causes this ... perhaps
Bart de Boer
@boekabart
@ that, I sometimes think Akka.Net needs a 'priority' system for actor mailboxes for such high-load cases - some actors are simply more time-critical than others. cc @Aaronontheweb
Stefano
@delfuria
Priority Mailboxes are already available
http://getakka.net/articles/actors/mailboxes.html
anthonyhawes
@anthonyhawes
I don't think priority mailboxes would solve the time-critical actor problem in high load cases. A priority scheduler would be needed to give those actors a larger "slice" of message processing
Vagif Abilov
@object
I am checking our cluster sharding event journal, and it has very small number of rows. And the snapshot store that we created for cluster sharding is always empty. So I wonder if there's any need for a snapshot store for cluster sharding. Can there ever be an attempt to create a snapshot? And is there ever need to cleanup the event journal for cluster sharding? Can it grow big (I suppose it doesn't have autocleanup).
Bartosz Sypytkowski
@Horusiath

@object if you don't have remember-entites flag set, cluster sharding won't produce too much events. Snapshot store will be used (by default) every 1000 consecutive sharding events stored, but this applies if shard coordinator will live long enough to produce that 1000 events within a single incarnation.

Also I hope to be able to finish DData-based cluster sharding soon (see akkadotnet/akka.net#3199). With it, event journals will no longer be needed.

It actually already works, but I need to fix tests.
wilddeveloperappears
@wilddeveloperappears
@Horusiath can I set them up programatically? Or does it have to be through HOCON config? I want to pass a function to the initial logger.
Vagif Abilov
@object
Thanks @Horusiath. We recently had weird experience with our cluster when nodes were not able to form a single cluster continuously logging "trying to register... but no acknowledgement" until we deleted whole sharding event journal (there were only about 80 rows in it). And if DData implementation is around the corner, we should not probably spend much time on the investigation of this failure.
Bartosz Sypytkowski
@Horusiath
@wilddeveloperappears if I remember right, you should be able to just subscribe to System.EventStream.Subscribe(actorRef, typeof(Akka.Event.LogEvent))
wilddeveloperappears
@wilddeveloperappears
I'm currently investigating doing hocon changes to solve the problem.
Bartosz Sypytkowski
@Horusiath
@object this sounds like an issue described by @zbynek001 (see: https://github.com/akkadotnet/akka.net/issues/3204#issuecomment-350274096) - regarding ddata, remember that PRs can take some time.
wilddeveloperappears
@wilddeveloperappears
I've retro-fitted NLog into a project I'm working on with hundreds of classes all separated out into projects. They all fed into one log file and now I have them filtering to several based on their namespace. Using Akka in this model is broken because the Akka loggers are in a separate namespace from the project they're logging for.
Bart de Boer
@boekabart
@delfuria That's something else - those allow messages to overtake each other. I'm talking about actors whose 'inbox' gets processed with priority over other actor's inboxes.
@anthonyhawes not a larger slice, but when the scheduler decides what to do next, it should 'prefer' to do the High Prio tasks, always.
(so it's important that there sometimes/often are none such)
This can be used in 'streams' (that are not built using Akka.Streams) as well to prevent having to use backpressure explicitly - just give the 'downstream' actors increasing priorities
Vagif Abilov
@object
@Horusiath sounds similar, but I didn't quite understand "On persistent plugin side, properly using Serialization.SerializeWithTransport method and passing transport context". Does this mean that there is a problem with SqlServer persistent plugin which we are using (and which apparently does not pass transport context)?
Bartosz Sypytkowski
@Horusiath
@object more a problem with how akka.net serialization handles actor refs (not visible in short living cases like remoting, but probably hurting persistence)
Maxim Cherednik
@maxcherednik
hey @Aaronontheweb shall I have a look into this: akkadotnet/akka.net#3208
scharada
@scharada
great framework. thank you guys. have one question though. i have read/heard about the consensus mechanism that can be achieved by akka.net. is there any sample that shows how it works and how it can be created ? would greatly appreciate it. @Aaronontheweb any hint ?
Alfredo Herrera
@alfredherr
Guys - is persistence sharding supported by the SQLite store as of version 1.3.2?
Bartosz Sypytkowski
@Horusiath
@alfredherr what do you mean by persistence sharding?
Bart de Boer
@boekabart
Sidenote: we've recently found out that Akka.Persistence.Sqlite is pretty bad at the SQL traffic pattern typical to Akka applications: INSERTs are reaaaallly slow
For example, a postgres spinned up on the same machine in docker outperformed it by like 10x
Bart de Boer
@boekabart
So think twice before settling on Sqlite ...
Saepul Ramdani
@blackclavus
I found bug on Udp, when client sending several data (different byte array), Udp.Received sometimes will give wrong bytes (the length is correct, but content isn't)
I'll write issue soon
Saepul Ramdani
@blackclavus
#3210
Alfredo Herrera
@alfredherr
@boekabart @Horusiath we are doing a POC of akka.net using file-based storage only, i.e. sqlite or rocksdb but we’re getting horrible performance with sqlite and i cant get rocksdb to work. I want rework/remodel it using sharding to have each actor define to what database it will store its state. that way i can have more than one database and hopefully get better throughput.
ForrestWang
@ForrestWang
@Aaronontheweb In Akka streams, Is it make sense to make the element immutable? In each stage, the transformation will generate a new element and downstream to the next stage, after the stream processing finished.
after the stream processing finished, no changes to the original data source.
Bart de Boer
@boekabart
@alfredherr sqlite seems to be really bad at INSERT performance, which is exactly the only thing Akka does when persisting. See https://blog.devart.com/increasing-sqlite-performance.html for some background and ideas to improve - we've actually tried most of these with very little effect
We've settled for SqlExpress or PostGres for now
HermansG
@HermansG
@boekabart You wrote you use OnBecome - that name suggests it is an virtual/abstract member of the base class. As far as I can seen it is not? Is such a functionality something that could become a feature?