Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jul 30 12:07
    blinker001 closed #5160
  • Jul 30 12:07
    blinker001 commented #5160
  • Jul 30 08:03
    blinker001 commented #5160
  • Jul 30 08:01
    blinker001 commented #5160
  • Jul 30 07:39
    blinker001 commented #5160
  • Jul 29 22:31
    blinker001 edited #5160
  • Jul 29 22:31
    blinker001 edited #5160
  • Jul 29 22:27
    blinker001 edited #5160
  • Jul 29 22:27
    blinker001 edited #5160
  • Jul 29 22:26
    blinker001 opened #5160
  • Jul 28 21:42
    Aaronontheweb milestoned #5159
  • Jul 28 21:42
    Aaronontheweb labeled #5159
  • Jul 28 21:42
    Aaronontheweb labeled #5159
  • Jul 28 21:42
    Aaronontheweb commented #5159
  • Jul 28 21:41
    Aaronontheweb labeled #5159
  • Jul 28 18:01
    Aaronontheweb commented #5157
  • Jul 28 18:01
    Aaronontheweb assigned #5157
  • Jul 28 18:01
    Aaronontheweb milestoned #5157
  • Jul 28 18:01
    Aaronontheweb labeled #5157
  • Jul 28 18:00
    Aaronontheweb labeled #5157
Joshua Garnett
@joshgarnett
Yeah
AndreSteenbergen
@AndreSteenbergen
@Aaronontheweb I understand, it can really crawl a lot, it seems changing the buffer size fixed the issue, don't know why or how, but the issue is gone, and a waiting list of 4 million pages is already absurd, so if the buffer would be full, I would have used too much memory already, so going back to a few 1000 is goog enough as well.
Aaron Stannard
@Aaronontheweb
is this the Akka.Remote buffer size or the crawler's stream setup?
because yeah, the Akka.Streams stuff needs to be tuned a bit - I think it's a bit too aggressive in terms of throttling down the crawl rate and level of concurrency at the moment
AndreSteenbergen
@AndreSteenbergen

Crawler stream set-up, with the smaller buffer I don't seem to get any issues.

I still need to tweak the config somewhat, on all node the standard config creates 2 download actors (per domain as I see it), with the maxConcurrency, making it distributed over multiple nodes will look a ddos attack in no time, don't want to bring down sites.

nathvi
@nathvi
I have a question with Akka Remote Deployment. If I deploy actor A from process A to process B, and process B stops or gets terminated, where does the state of actor A reside?
Aaron Stannard
@Aaronontheweb
actor A lives physically in process B
so actor A's state will be destroyed if B is terminated
you'd want to persist that state to Akka.Persistence or replicate it elsewhere
to make it recoverable or otherwise available
Lutando Ngqakaza
@Lutando
when doing cross cluster messaging do my messages have to be mutable?
Aaron Stannard
@Aaronontheweb
nope
just need to be serializable
they should always be immutable ideally
Lutando Ngqakaza
@Lutando
yeah
I am trying to figure out why a EventStream.Publish object is coming back with null fields where those fields are pocos with private setters
everything else (mutatable fields) are fine
if i make them private set they appear null too
will dig further
Aaron Stannard
@Aaronontheweb
using the built-in JSON serializer I take it?
Lutando Ngqakaza
@Lutando
i beleive we are using the defaults for serialization
Aaron Stannard
@Aaronontheweb
ok, got it
the JSON serializer just may not be able to figure out how to populate those values since it relies on conventions / magic IMHO to do it
but imho
Lutando Ngqakaza
@Lutando
oh so between clusters it uses JSON and not Hyperion as default?
Aaron Stannard
@Aaronontheweb
yeah Hyperion isn't used as a default anywhere at the moment
if you're serious about using Akka.Remote or Akka.Cluster in production though
you're better off going with hand-rolled serialization using something that has an explicit scheme like Google.Protobuf
better performance, first and foremost
but more importantly, this type of issue doesn't happen
since you can explicitly control serialization
also makes it easier to version messages on the wire too
Lutando Ngqakaza
@Lutando
we will probably end up there, right now were doing a "large" proof of concept
Aaron Stannard
@Aaronontheweb
we ship with JSON.NET / Hyperion in the future as a default because it's easy to get up and running
but those polymorphic serializers aren't perfect
might have to give them some hints on how to do their job
Lutando Ngqakaza
@Lutando
i see
Jesse Connor
@jesseconnr
I'm having an issue with message delivery in a setup remoting between a windows service and an asp.net site. I'm using an aggreggator to send a large amount of messages to keep the size small, but out of 3500 messages only 1700 or so are making it. In smaller quantities, when there's only a few hundred there's no problems.
My agreggator and the hocon config for the two is here: https://gist.github.com/jesseconnr/256da997b49e38026c32f4bc2987adf5
Aaron Stannard
@Aaronontheweb
are you hitting your receive timeout?
and shutting down?
Jesse Connor
@jesseconnr
Both are on the same machine. I get errors like this Error while decoding incoming Akka PDU of length 28885 but I'm not sure what to make of it.
Aaron Stannard
@Aaronontheweb
ah shit
that's a serialization problem
Jesse Connor
@jesseconnr
Oh the 30 seconds? No I don't think so.
Aaron Stannard
@Aaronontheweb
this is with Hyperion
do you get that issue without it?
and is this on Linux?
Jesse Connor
@jesseconnr
It's on windows.
Aaron Stannard
@Aaronontheweb
so we've had reports of DotNetty fucking up its length-frame encoding