Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Oct 21 18:54
    peirens-bart closed #4095
  • Oct 21 18:46
    Aaronontheweb closed #5310
  • Oct 21 18:16

    Aaronontheweb on dev

    Bump Akka.MultiNode.TestAdapter… (compare)

  • Oct 21 18:16

    Aaronontheweb on nuget

    (compare)

  • Oct 21 18:16
    Aaronontheweb closed #5335
  • Oct 21 16:10

    dependabot[bot] on nuget

    (compare)

  • Oct 21 16:10
    dependabot[bot] closed #5299
  • Oct 21 16:10
    dependabot[bot] commented #5299
  • Oct 21 16:10
    dependabot[bot] labeled #5336
  • Oct 21 16:10
    dependabot[bot] opened #5336
  • Oct 21 16:10

    dependabot[bot] on nuget

    Bump Google.Protobuf from 3.17.… (compare)

  • Oct 21 16:10
    dependabot[bot] labeled #5335
  • Oct 21 16:10

    dependabot[bot] on nuget

    Bump Akka.MultiNode.TestAdapter… (compare)

  • Oct 21 16:10
    dependabot[bot] opened #5335
  • Oct 21 13:18

    Aaronontheweb on dev

    Fix DefaultResizer for suspende… (compare)

  • Oct 21 13:18
    Aaronontheweb closed #5333
  • Oct 21 13:18
    Aaronontheweb closed #5327
  • Oct 21 13:17
    Aaronontheweb labeled #5334
  • Oct 21 13:17
    Aaronontheweb labeled #5334
  • Oct 21 13:17
    Aaronontheweb opened #5334
AndreSteenbergen
@AndreSteenbergen
that's the weird thing, or do I need to do that
AndreSteenbergen
@AndreSteenbergen
Thanks, just forced a mailbox using the withMailbox as parameter, works as a charm.

I do have 1 thing I don't know if it's a bug or something else, I occasionally see this error, after that the flow just stops ... I am using the example of the web crawler

[ERROR][11-4-2018 10:47:00][Thread 0018][akka://crawler/remote/akka.tcp/crawler@127.0.0.1:61662/user/api/XXXXXXX/coordinators/c1/StreamSupervisor-1/Flow-0-1-actorRefSource] The LinkedList is empty.
Cause: System.InvalidOperationException: The LinkedList is empty.
at System.Collections.Generic.LinkedList1.RemoveLast() at Akka.Streams.Implementation.ActorRefSourceActor1.ReceiveElement(T message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)

I can't seem to pinpoint the problem. And I don't know is it's a bug in my code or akka.net. I did tweak the numbers a bit, to not let websites crash. The code example really is a crawler of doom ;)
AndreSteenbergen
@AndreSteenbergen
250 per minute with 2 maxconcurrent.
        var htmlFlow = Flow.Create<PageToCrawl>()
            .Throttle(job.Config.ThottleAmount, job.Config.PerTimeUnit, job.Config.ThottleAmount, ThrottleMode.Shaping)
            .Via(DownloadFlow.ProcessHtmlDownloadFor(MaxConcurrentDownloads, job, pageRequester));
AndreSteenbergen
@AndreSteenbergen
When that happens the entire actor won't receive anything anymore
or the stream at least
AndreSteenbergen
@AndreSteenbergen
I have a near infinite buffer: var source = Source.ActorRef<PageToCrawl>(int.MaxValue, OverflowStrategy.DropTail);
AndreSteenbergen
@AndreSteenbergen
I did find RemoveLast is called by DropTail, but I know for sure I have not reached int.MaxValue items in the buffer.
Marc Piechura
@marcpiechura
@AndreSteenbergen could you try to reproduce it in a simpler example?
AndreSteenbergen
@AndreSteenbergen
I tried ....
It isn't actually that hard, it's also doesn't happen evry time
30% of the time I have a crawl of about 600 pages
I have also tried debug messages, but the actors are quite chatty in debug mode. Can't seem to find any useful data
AndreSteenbergen
@AndreSteenbergen
Or can it be as simple of choosing a smaller buffer e.g. 2048 * 2048? 4 million in that queue is quite a lot I guess ...
AndreSteenbergen
@AndreSteenbergen
I have test run my crawl 5 times now, still no issue ... Could Buffer.IsFull be a bit problematic with queues of size int.MaxValue?
Marc Piechura
@marcpiechura
Not sure, I don’t know how the web crawler example looks like, maybe @Aaronontheweb knows more
Joshua Garnett
@joshgarnett
Are there any known performance issues I should watch out for when running a debug vs a release build? We are seeing significant performance hits on debug builds, more then I would have expected.
Aaron Stannard
@Aaronontheweb
@AndreSteenbergen I've had the WebCrawler sample, as it currently stands, process 20GB+ of pages before
didn't have any issues really
@joshgarnett other than the usual stuff in the CLR that we don't control, none that I am aware
the Akka.NET binaries we ship on NuGet don't have any conditional code that checks for the presence of an attached debugger or anything
Joshua Garnett
@joshgarnett
hmm I might have some profiling in my future, now to find a profiler that works well on OSX for .net core
Aaron Stannard
@Aaronontheweb
I assume you're not using a local build of Akka.NET or anything, right?
Joshua Garnett
@joshgarnett
Akka.Persistence.MySQL is being built locally
otherwise using some nuget packages
but that should never affect you if you're using the NuGet packages
only comes into play when running Akka.NET locally in debug
Joshua Garnett
@joshgarnett
interseting
I’m not seeing any ifdefs for the Akka.Persistence.MySQL build
Aaron Stannard
@Aaronontheweb
I assume what you're seeing is a drop in your entire app
performance, right?
Joshua Garnett
@joshgarnett
Yeah
AndreSteenbergen
@AndreSteenbergen
@Aaronontheweb I understand, it can really crawl a lot, it seems changing the buffer size fixed the issue, don't know why or how, but the issue is gone, and a waiting list of 4 million pages is already absurd, so if the buffer would be full, I would have used too much memory already, so going back to a few 1000 is goog enough as well.
Aaron Stannard
@Aaronontheweb
is this the Akka.Remote buffer size or the crawler's stream setup?
because yeah, the Akka.Streams stuff needs to be tuned a bit - I think it's a bit too aggressive in terms of throttling down the crawl rate and level of concurrency at the moment
AndreSteenbergen
@AndreSteenbergen

Crawler stream set-up, with the smaller buffer I don't seem to get any issues.

I still need to tweak the config somewhat, on all node the standard config creates 2 download actors (per domain as I see it), with the maxConcurrency, making it distributed over multiple nodes will look a ddos attack in no time, don't want to bring down sites.

nathvi
@nathvi
I have a question with Akka Remote Deployment. If I deploy actor A from process A to process B, and process B stops or gets terminated, where does the state of actor A reside?
Aaron Stannard
@Aaronontheweb
actor A lives physically in process B
so actor A's state will be destroyed if B is terminated
you'd want to persist that state to Akka.Persistence or replicate it elsewhere
to make it recoverable or otherwise available
Lutando Ngqakaza
@Lutando
when doing cross cluster messaging do my messages have to be mutable?
Aaron Stannard
@Aaronontheweb
nope
just need to be serializable
they should always be immutable ideally
Lutando Ngqakaza
@Lutando
yeah
I am trying to figure out why a EventStream.Publish object is coming back with null fields where those fields are pocos with private setters
everything else (mutatable fields) are fine