Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 07 12:19
    jeremydmiller labeled #669
  • Jul 07 12:19
    jeremydmiller labeled #668
  • Jul 07 12:19
    jeremydmiller labeled #667
  • Jul 07 12:19
    jeremydmiller labeled #666
  • Jul 07 12:19
    jeremydmiller labeled #665
  • Jul 07 12:11
    jeremydmiller commented #668
  • Jul 07 12:02
    jeremydmiller demilestoned #644
  • Jul 07 12:02
    jeremydmiller demilestoned #629
  • Jul 07 12:02
    jeremydmiller demilestoned #627
  • Jul 07 12:02
    jeremydmiller demilestoned #618
  • Jul 07 12:02
    jeremydmiller demilestoned #617
  • Jul 07 12:02
    jeremydmiller demilestoned #616
  • Jul 07 12:02
    jeremydmiller demilestoned #612
  • Jul 07 12:02
    jeremydmiller demilestoned #610
  • Jul 07 12:02
    jeremydmiller demilestoned #608
  • Jul 07 12:02
    jeremydmiller demilestoned #607
  • Jul 07 12:02
    jeremydmiller demilestoned #602
  • Jul 07 12:02
    jeremydmiller demilestoned #601
  • Jul 07 12:02
    jeremydmiller demilestoned #600
  • Jul 07 12:02
    jeremydmiller demilestoned #599
Mark Warpool
@CodingGorilla
Looking through the Jasper source, I can't see where it sets the MaxDegreeOfParallelism to anything other than what the default value for ExecutionDataflowBlockOptions is based on the default constructor.
The issue appears to be that it runs multiple handlers in parallel, but if I have 5 messages of one handler type, it runs those in sequence. Which lines up with the Dataflow library says I should expect (unless I'm missing something)
There's a sample where it's set to 5, but I don't see anything in Jasper itself that actually configures it
Jeremy D. Miller
@jeremydmiller
When you configure any listener, there’s a fluent interface that let’s you hcange the block options for that listener
I’ll look up the code after dinner, but the default in Jasper is 5 unless I’m crazy.
Look for MaximumThreads() in this: http://jasperfx.github.io/documentation/local/
Mark Warpool
@CodingGorilla
Yea, I see that and was aware of that. But as far as I can tell the Endpoint.ExecutionOptions (for which the .MaximumThreads() operates on) is only constructed by using the default constructor of the ExecutionDataflowBlockOptions and nowhere is the parallelism ever changed from what the defaults are.
Jeremy D. Miller
@jeremydmiller
Yes, and the MaximumThreads() method modifies that, right?
Mark Warpool
@CodingGorilla
Yes, but unless you call MaximumThreads() it will default to 1.
essentially meaning it defaults to .Sequential()
Jeremy D. Miller
@jeremydmiller
Oops, yeah. It used to be 5, but that must have gotten lost along the way
Have you set MaximumThreads() then? And that’s completely orthogonal to which handlers are running
Mark Warpool
@CodingGorilla
I have an update to set .MaximumThreads(-1) (unbounded) but I have to wait for it to empty out it's work queue so I don't stop it in the middle of processing a message.
Jeremy D. Miller
@jeremydmiller
You can’t change that when the app is running, only upfront
Mark Warpool
@CodingGorilla
Yea, I need to stop the app and redeploy, but it's a data importer, so if I stop it in the middle of a message, I will likely end up with duplicated imports.
Jeremy D. Miller
@jeremydmiller
gotcha
Mark Warpool
@CodingGorilla
But at least I know I'm on the right track, I was scratching my head because I agreed with you in that I expected it to run the messages with some degree of parallelism.
Would you be opposed to a PR that defaults that to -1, which means "Unbounded" or more like "bounded only by system resources"?
Or maybe just an extension method called .Unbounded() so that it doesn't change the default behavior (maybe until a 2.0 release)
Jeremy D. Miller
@jeremydmiller
Sure. I’ll switch to Jasper soon. Or make it “number of cores” or something.
Mark Warpool
@CodingGorilla
@jeremydmiller If I need to defer handling a message for some reason, can I do something like:
public async Task<object> Handle(MyMessage message)
{
   if(this.Defer)
      return new Envelope(message).ScheduleAt(later);

   ... do normal handling ...

   return null;
}
Jeremy D. Miller
@jeremydmiller
yes
Mark Warpool
@CodingGorilla
cool, ty
Mark Warpool
@CodingGorilla
@jeremydmiller So I have an Azure App service, which can scale out instances to handle load. It uses jasper to queue up "events" to handle as data is inserted. We had a huge import run for a couple of days and what I noticed is that the jasper_incoming_messages has over 7500 messages queued up and just sitting there, they don't seem to be clearing out at all. I stopped and started the app, and the owner_id seemed to change, so it seems like Jasper knows the messages are there. Any ideas/thoughts? (All messages are set to publish locally, no transports)
Jeremy D. Miller
@jeremydmiller
Maybe database locks so nothing can take ownership? You should see worlds of logging about messages being recovered and processed, so anything coming through logging?
Possibly a poison pill if you set everything to single threaded?
Mark Warpool
@CodingGorilla
I don't see anything in the logs, but it's a production environment so only warnings and errors would appear. The max concurrency is - 1.
Jeremy D. Miller
@jeremydmiller
No exceptions whatsoever? And is there any way it's a poison pill problem, because there's no command timeout enforced by Jasper itself
Mark Warpool
@CodingGorilla
I don't think so, none of them should actually even do any work except a simple database load, because the data is old. And no, no exceptions are being logged.
Jeremy D. Miller
@jeremydmiller
When you restart, do you see the owner_id getting set to zero, then nothing? Or some of them being reset to something else?
Mark Warpool
@CodingGorilla
I see some of them being reset to something else, although after a long period I saw the "something else" id drop from a count of 200 to 199, which seems to lend some credence to your "poison pill" idea. I've fired up a local instance in the debugger and connected to production, but it doesn't seem to be attempting to recover any of the incoming messages. With the local queue set to durable, should it?
Jeremy D. Miller
@jeremydmiller
You just need the message durability turned on, but it's also been 2+ years since I've been into that code much
Mark Warpool
@CodingGorilla
ok, I'll keep digging around
Mark Warpool
@CodingGorilla
being a postgres database, it looks like the serialized message is stored binary, do you know if there's a way to see the messages?
Jeremy D. Miller
@jeremydmiller
No, but that sounds like a good idea for Jasper vNext :(
It's using some really old RhinoQueues format for writing headers and the message body in one byte stream.
Mark Warpool
@CodingGorilla
@jeremydmiller You were dead on with that poison pill idea, it just wasn't the event that I thought it was. :+1:
Jeremy D. Miller
@jeremydmiller
With Marten v4 just about to be done, I might need to start a longer conversation about the future of Jasper. Something along the lines of:
  1. Throw in the towel
  2. Just make it an in memory mediator that's far beyond MediatR in functionality
  3. Make it be an add-on to MassTransit so you get Jasper style handlers and discovery on top of MassTransit's plumbing
  4. Keep evolving Jasper as a standalone, MassTransit/NServiceBus/MediatR/Brighter competitor
And I think I want to finally release the Jasper.HTTP stuff regardless
Mark Warpool
@CodingGorilla
What are you throwing the towel in on?
all of the "transport stuff"?
Jeremy D. Miller
@jeremydmiller
I haven't made any decisions yet, but 1.) would be giving up on everything. 2.) would be throwing away the transport stuff.
If 4.), then it's important to have MassTransit and NServiceBus compatibility -- which I don't think is really going to be that hard
Mark Warpool
@CodingGorilla
Oh, I read those as a to-do list rather than individual options for how to proceed. =) Do you have any leanings?
Jeremy D. Miller
@jeremydmiller
Changes hour to hour:)
the pandemic, Marten v4, and my work being so dreadful last year just knocked me out of Jasper dev for awhile.
Mark Warpool
@CodingGorilla
I totally understand, I've tried a couple of times to go in there (Jasper) and start upgrading it to .NET 5 and it's a beast of a project..
Jeremy D. Miller
@jeremydmiller
It's not that big
Mark Warpool
@CodingGorilla
it just has a lot of moving parts, I got hung up because (I think) of the dependencies on Marten and Npgsql, the core project(s) weren't that big of a deal.