Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 18:29

    jeremydmiller on master

    removing the now nonexistent Ja… (compare)

  • 18:25

    jeremydmiller on master

    turned off the tests in CI. #sa… (compare)

  • 18:12

    jeremydmiller on master

    moved CoreTests to Jasper.Testi… combined CoreTests and Messagin… turning unit tests back on for … (compare)

  • 16:48
    jeremydmiller closed #153
  • 16:48
    jeremydmiller commented #153
  • 16:47
    jeremydmiller closed #470
  • 16:47
    jeremydmiller commented #470
  • 16:46
    jeremydmiller closed #546
  • 16:46
    jeremydmiller commented #546
  • 16:45
    jeremydmiller closed #547
  • 16:45
    jeremydmiller commented #547
  • 16:44
    jeremydmiller closed #549
  • 16:44

    jeremydmiller on master

    moved the JasperHttp code to it… (compare)

  • 16:19

    jeremydmiller on master

    correcting the artifacts path f… (compare)

  • 16:13

    jeremydmiller on master

    deleting the unused benchmarks (compare)

  • 15:58

    jeremydmiller on master

    installing netcore 3 in appveyo… (compare)

  • 15:56

    jeremydmiller on master

    bumping to 0.9.15 (compare)

  • 15:47
    jeremydmiller closed #545
  • 15:47
    jeremydmiller commented #545
  • 15:46

    jeremydmiller on master

    moving functionality for serial… thinning down ModelReader more introducing new IReader/WriterS… and 24 more (compare)

Jeremy D. Miller
@jeremydmiller
You’re using Postgres or Sql Server for the message persistence?
Mark Warpool
@CodingGorilla
Postgres via Jasper.Persistence.Marten
Jeremy D. Miller
@jeremydmiller
Can we do a zoom meeting or google hangout some day on this?
You’re running into a lot of things I don’t understand
Mark Warpool
@CodingGorilla
sure
I work from home, so I can make myself available whenever is good for you, except next thursday or friday
Jeremy D. Miller
@jeremydmiller
Monday afternoon work maybe?
Mark Warpool
@CodingGorilla
yea, that would be fine
Mark Warpool
@CodingGorilla
Man, I've been trying to figure this out all night, it has to be some weird sort of concurrency/timing issue, if I slow it down enough I can make it go away. But here is smidge of what I see in the Output when it actually happens
Processing: $Envelope #016cd0d2-9646-4d42-8579-f9c698867afb (UpdateFeatureSubscriptions) from Api@CGPC to loopback://durable/ on default
[... snipped noise ...]
Jasper.Transports:Information: Recovered 1 incoming envelopes from storage
Processing: $Envelope #016cd0d2-9646-4d42-8579-f9c698867afb from Api@CGPC to loopback://durable/ on default
[... snipped more noise ...]
Jasper.Messages:Information: Successfully processed message UpdateFeatureSubscriptions#016cd0d2-9646-4d42-8579-f9c698867afb from loopback://retries/
[... snipped more noise ...]
Jasper.Messages:Information: Successfully processed message UpdateFeatureSubscriptions#016cd0d2-9646-4d42-8579-f9c698867afb from loopback://retries/
Jay Stevens
@jediwarpraptor
This message was deleted
Mark Warpool
@CodingGorilla
The Processing: $Envelope #..." is a diagnostic log message I added to theWorkerQueue.AddQueue`ActionBlock<Envelop> anonymous method
Jay Stevens
@jediwarpraptor
This message was deleted
Mark Warpool
@CodingGorilla
So the one thing I noticed is that the first time I see the "Processing message", there is no other Jasper logging before it. But the second time you see the "Recovered 1 incoming envelopes from storage" message. This leads me to believe that it some how queues the job to run immediately, but then the RecoverIncomingMessages kicks in and grabs it also? But I'll be damned to hell if I can figure out where that's happening.
Jeremy D. Miller
@jeremydmiller
It it the same message Guid?
Mark Warpool
@CodingGorilla
yea, that sample I quoted above is real, the id is: 016cd0d2-9646-4d42-8579-f9c698867afb
Jeremy D. Miller
@jeremydmiller
It really shouldn’t be possible for it to be doing that if the same node is running the whole time
I’d love to see how you’re making that happen. This is all from posting to a local, persistent queue, right?
And you’re not stopping and starting anything in between somehow, right?
Mark Warpool
@CodingGorilla
This is all running on my local development machine. Yea, it's all from a call to ICommandBus.EnqueueDurably()
nope, like I said I have to remove all the breakpoints except inside the handler, or the behavior goes away
Jeremy D. Miller
@jeremydmiller
Is there something from the routing rules that’s routing it to both queues?
Hey, I’ve got to turn in, but I’ll check this when I can tomorrow.
Mark Warpool
@CodingGorilla
Not as far as I can tell, I really only have the local queue, I'll keep digging around until I'm ready to pass out myself. Good night :)
Jeremy D. Miller
@jeremydmiller
ttyl
Mark Warpool
@CodingGorilla
So a couple of observations:

First, this message:

Jasper.Messages:Information: Successfully processed message UpdateFeatureSubscriptions#016cd11a-6c35-483d-a028-fbfe4a5535f4 from loopback://retries/

Seems misleading, it seems like after a message is processed it always calls MessageLogger.MessageSucceeded() which is hard coded to report the "loopback://retries/" URL (which is called ReplyUri??), no matter what. The other logging I added seems to indicate it's actually coming from "loopback://durable/"
Second, I changed the JasperOptions.ScheduledJobs.FirstExecution and JasperOptions.ScheduledJobs.PollingTime, which seems to control the durability agent's queue processing. I set them both to 2 minutes (default of 0 seconds and 5 seconds respectively), and the problem seems to go away.
Mark Warpool
@CodingGorilla
But along with the problem going away, I also notice that the message is no longer being picked up by the RecoverIncomingMessages queue from the durability agent. But I still cannot figure out where the actual execution comes from.
I'm sure I'll be back, banging my head on this wall in the morning, I hate a puzzle I can't solve =D
Mark Warpool
@CodingGorilla
Oh, I think I figured it out, if I understand incoming messages structure properly, an OwnerId of 0 means "AnyNode". EnqueueDurably() calls down into persistOrSend, which since there isn't an ongoing transaction, calls Envelope.Send(). This in turn calls the DurabilityLoopbackSendingAgent.StoreAndForward().
Mark Warpool
@CodingGorilla
This is where the problem is. .StoreAndForward() initially saves the message as an incoming message with an OwnerId of 0 (= AnyNode), and then immediately throws it into it's WorkerQueue, which will start running the handler regardless of the "owning node". Then, since the RecoverIncomingMessages fires every 5 seconds by default, assuming the handler takes longer than 5 seconds to run, it will see the message sitting in the incoming messages table, and "recover it" and run it again.
BOOM, problem identified. Gotta sleep now. =)
Jeremy D. Miller
@jeremydmiller
Yup, that would do that. Awfully good catch.
Jeremy D. Miller
@jeremydmiller
And a one line code fix:/
Mark Warpool
@CodingGorilla
Is the fix to just make.StoreAndForward() save it with the current node id?
Jeremy D. Miller
@jeremydmiller
Yes
Mark Warpool
@CodingGorilla
Makes sense...
Mark Warpool
@CodingGorilla
Would you like me to prepare a pull request? Or have you got it?
Jeremy D. Miller
@jeremydmiller
I’ll take a PR, or I’m going to be back in Jasper for other reasons in the next week I think
Mark Warpool
@CodingGorilla
#532 Is ready to go
Jeremy D. Miller
@jeremydmiller
:thumbsup:
Got it
or will very soon
Mark Warpool
@CodingGorilla
Thanks, we have a big release of a different product next week, so I have a little breathing room on this particular issue, so no rush.
That and I put a bunch of locking on the handler to make sure it can't run twice at the same time for the short term =)
Svein Ellingsen
@lovmoen
Is there a way to perform a remote async "request/response" in the same way as Invoke<TResponse>()?
Jeremy D. Miller
@jeremydmiller
I don’t remember if that got yanked out or not, but there used to be an Request() method for that. I think I eliminated that in the thinking that you’d just use HTTP. That was some significant complexity
Svein Ellingsen
@lovmoen
OK, thanks for your quick response! I'll just have to figure a way around that then.