Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Oct 18 22:26
    to11mtm commented #5203
  • Oct 18 22:00
    Aaronontheweb commented #5203
  • Oct 18 21:58
    Aaronontheweb commented #5203
  • Oct 18 21:56
    to11mtm commented #5203
  • Oct 18 21:22
    Arkatufus synchronize #5326
  • Oct 18 20:52
    Arkatufus commented #5326
  • Oct 18 20:52
    Arkatufus converted_to_draft #5326
  • Oct 18 20:52
    Arkatufus synchronize #5326
  • Oct 18 20:51
    Arkatufus edited #5326
  • Oct 18 20:49
    Arkatufus opened #5326
  • Oct 18 20:47
    Arkatufus commented #5325
  • Oct 18 20:32
    Aaronontheweb commented #5325
  • Oct 18 20:26
    Arkatufus commented #5325
  • Oct 18 20:21
    Aaronontheweb commented #5325
  • Oct 18 20:02
    Arkatufus commented #5325
  • Oct 18 19:59
    Aaronontheweb commented #5325
  • Oct 18 19:57
    Arkatufus commented #5325
  • Oct 18 18:19
    Aaronontheweb commented #5325
  • Oct 18 18:10
    Arkatufus commented #5325
  • Oct 18 17:09
    SeanKilleen commented #5320
AndreSteenbergen
@AndreSteenbergen
I have also tried debug messages, but the actors are quite chatty in debug mode. Can't seem to find any useful data
AndreSteenbergen
@AndreSteenbergen
Or can it be as simple of choosing a smaller buffer e.g. 2048 * 2048? 4 million in that queue is quite a lot I guess ...
AndreSteenbergen
@AndreSteenbergen
I have test run my crawl 5 times now, still no issue ... Could Buffer.IsFull be a bit problematic with queues of size int.MaxValue?
Marc Piechura
@marcpiechura
Not sure, I don’t know how the web crawler example looks like, maybe @Aaronontheweb knows more
Joshua Garnett
@joshgarnett
Are there any known performance issues I should watch out for when running a debug vs a release build? We are seeing significant performance hits on debug builds, more then I would have expected.
Aaron Stannard
@Aaronontheweb
@AndreSteenbergen I've had the WebCrawler sample, as it currently stands, process 20GB+ of pages before
didn't have any issues really
@joshgarnett other than the usual stuff in the CLR that we don't control, none that I am aware
the Akka.NET binaries we ship on NuGet don't have any conditional code that checks for the presence of an attached debugger or anything
Joshua Garnett
@joshgarnett
hmm I might have some profiling in my future, now to find a profiler that works well on OSX for .net core
Aaron Stannard
@Aaronontheweb
I assume you're not using a local build of Akka.NET or anything, right?
Joshua Garnett
@joshgarnett
Akka.Persistence.MySQL is being built locally
otherwise using some nuget packages
but that should never affect you if you're using the NuGet packages
only comes into play when running Akka.NET locally in debug
Joshua Garnett
@joshgarnett
interseting
I’m not seeing any ifdefs for the Akka.Persistence.MySQL build
Aaron Stannard
@Aaronontheweb
I assume what you're seeing is a drop in your entire app
performance, right?
Joshua Garnett
@joshgarnett
Yeah
AndreSteenbergen
@AndreSteenbergen
@Aaronontheweb I understand, it can really crawl a lot, it seems changing the buffer size fixed the issue, don't know why or how, but the issue is gone, and a waiting list of 4 million pages is already absurd, so if the buffer would be full, I would have used too much memory already, so going back to a few 1000 is goog enough as well.
Aaron Stannard
@Aaronontheweb
is this the Akka.Remote buffer size or the crawler's stream setup?
because yeah, the Akka.Streams stuff needs to be tuned a bit - I think it's a bit too aggressive in terms of throttling down the crawl rate and level of concurrency at the moment
AndreSteenbergen
@AndreSteenbergen

Crawler stream set-up, with the smaller buffer I don't seem to get any issues.

I still need to tweak the config somewhat, on all node the standard config creates 2 download actors (per domain as I see it), with the maxConcurrency, making it distributed over multiple nodes will look a ddos attack in no time, don't want to bring down sites.

nathvi
@nathvi
I have a question with Akka Remote Deployment. If I deploy actor A from process A to process B, and process B stops or gets terminated, where does the state of actor A reside?
Aaron Stannard
@Aaronontheweb
actor A lives physically in process B
so actor A's state will be destroyed if B is terminated
you'd want to persist that state to Akka.Persistence or replicate it elsewhere
to make it recoverable or otherwise available
Lutando Ngqakaza
@Lutando
when doing cross cluster messaging do my messages have to be mutable?
Aaron Stannard
@Aaronontheweb
nope
just need to be serializable
they should always be immutable ideally
Lutando Ngqakaza
@Lutando
yeah
I am trying to figure out why a EventStream.Publish object is coming back with null fields where those fields are pocos with private setters
everything else (mutatable fields) are fine
if i make them private set they appear null too
will dig further
Aaron Stannard
@Aaronontheweb
using the built-in JSON serializer I take it?
Lutando Ngqakaza
@Lutando
i beleive we are using the defaults for serialization
Aaron Stannard
@Aaronontheweb
ok, got it
the JSON serializer just may not be able to figure out how to populate those values since it relies on conventions / magic IMHO to do it
but imho
Lutando Ngqakaza
@Lutando
oh so between clusters it uses JSON and not Hyperion as default?
Aaron Stannard
@Aaronontheweb
yeah Hyperion isn't used as a default anywhere at the moment
if you're serious about using Akka.Remote or Akka.Cluster in production though
you're better off going with hand-rolled serialization using something that has an explicit scheme like Google.Protobuf
better performance, first and foremost
but more importantly, this type of issue doesn't happen