Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 07 12:19
    jeremydmiller labeled #669
  • Jul 07 12:19
    jeremydmiller labeled #668
  • Jul 07 12:19
    jeremydmiller labeled #667
  • Jul 07 12:19
    jeremydmiller labeled #666
  • Jul 07 12:19
    jeremydmiller labeled #665
  • Jul 07 12:11
    jeremydmiller commented #668
  • Jul 07 12:02
    jeremydmiller demilestoned #644
  • Jul 07 12:02
    jeremydmiller demilestoned #629
  • Jul 07 12:02
    jeremydmiller demilestoned #627
  • Jul 07 12:02
    jeremydmiller demilestoned #618
  • Jul 07 12:02
    jeremydmiller demilestoned #617
  • Jul 07 12:02
    jeremydmiller demilestoned #616
  • Jul 07 12:02
    jeremydmiller demilestoned #612
  • Jul 07 12:02
    jeremydmiller demilestoned #610
  • Jul 07 12:02
    jeremydmiller demilestoned #608
  • Jul 07 12:02
    jeremydmiller demilestoned #607
  • Jul 07 12:02
    jeremydmiller demilestoned #602
  • Jul 07 12:02
    jeremydmiller demilestoned #601
  • Jul 07 12:02
    jeremydmiller demilestoned #600
  • Jul 07 12:02
    jeremydmiller demilestoned #599
Jeremy D. Miller
@jeremydmiller
I won’t have bandwidth for Jasper anytime soon, but OpenTelemetry support and perf optimization is the last 2 things for a 2.0
Jarrod Johnson
@JarrodJ83
I'm doing some work with open telemetry and pulsar.client soon. If all goes well I would be open to trying to implement in jasper as well
Nico Sap
@NicoJuicy
Am i correct to assume Jasper is a good fit for using as a Saga ( that can be persisted) and plumbed with Mediator? ( since it's a monolith for now)
Jeremy D. Miller
@jeremydmiller
Yes, and I see your question in the Marten room. Jasper supports the Saga pattern w/ Marten being one of the persistence strategies. Jasper is also a superset of the MediatR functionality
Nico Sap
@NicoJuicy
Yeah indeed. Okay, thx
I remembered the repo of Oskardudycz, checked his repository and saw something very similar to what i wanted. So i posted it on 2 channels since i didn't saw him post here, perhaps not the best idea, sorry
Jeremy D. Miller
@jeremydmiller
No worries, I follow both rooms:)
ndcomplete
@ndcomplete
@jeremydmiller I have a set of merge requests coming in: one for Oakton.AspNetCore, then one for Jasper following. I'm going to wait for the Oakton one to be finished (if you decide to approve it), as it is a dependency for the Jasper change. If you'd like, I'd be happy to push up a branch with the majority of the Jasper changes required, with a stub for the Oakton update, so you can see the intent of the changes.
ndcomplete
@ndcomplete
Here's the jasper changeset. Note that in a further changeset I reverted some of the unnecessary changes (temporary oakton version change and language change):
ndcomplete/jasper@e6e4557
Mark Warpool
@CodingGorilla
@jeremydmiller I have an app that's using 1.1.4 and seems to be only running one handler at a time. I can see that it has 4 messages queued up in the incoming messages table, but my telemetry only shows it running one handler. Any quick thoughts on things to check to figure out why?
Mark Warpool
@CodingGorilla
Wondering if I need to explicitly set the MaximumThreads() to something greater than one, it looks like the ExecutionDataflowBlockOptions.MaxDegreeOfParallelism defaults to 1.
Jeremy D. Miller
@jeremydmiller
It should be defaulting to 5 in Jasper (super arbitrary number), but that wouldn’t impact how many handlers are known in the system. Can you resolve HandlerGraph and see if there’s more than the one handler?
It would super help right now if I had Oakton Describe() commands in there for Jasper :/
Mark Warpool
@CodingGorilla
Looking through the Jasper source, I can't see where it sets the MaxDegreeOfParallelism to anything other than what the default value for ExecutionDataflowBlockOptions is based on the default constructor.
The issue appears to be that it runs multiple handlers in parallel, but if I have 5 messages of one handler type, it runs those in sequence. Which lines up with the Dataflow library says I should expect (unless I'm missing something)
There's a sample where it's set to 5, but I don't see anything in Jasper itself that actually configures it
Jeremy D. Miller
@jeremydmiller
When you configure any listener, there’s a fluent interface that let’s you hcange the block options for that listener
I’ll look up the code after dinner, but the default in Jasper is 5 unless I’m crazy.
Look for MaximumThreads() in this: http://jasperfx.github.io/documentation/local/
Mark Warpool
@CodingGorilla
Yea, I see that and was aware of that. But as far as I can tell the Endpoint.ExecutionOptions (for which the .MaximumThreads() operates on) is only constructed by using the default constructor of the ExecutionDataflowBlockOptions and nowhere is the parallelism ever changed from what the defaults are.
Jeremy D. Miller
@jeremydmiller
Yes, and the MaximumThreads() method modifies that, right?
Mark Warpool
@CodingGorilla
Yes, but unless you call MaximumThreads() it will default to 1.
essentially meaning it defaults to .Sequential()
Jeremy D. Miller
@jeremydmiller
Oops, yeah. It used to be 5, but that must have gotten lost along the way
Have you set MaximumThreads() then? And that’s completely orthogonal to which handlers are running
Mark Warpool
@CodingGorilla
I have an update to set .MaximumThreads(-1) (unbounded) but I have to wait for it to empty out it's work queue so I don't stop it in the middle of processing a message.
Jeremy D. Miller
@jeremydmiller
You can’t change that when the app is running, only upfront
Mark Warpool
@CodingGorilla
Yea, I need to stop the app and redeploy, but it's a data importer, so if I stop it in the middle of a message, I will likely end up with duplicated imports.
Jeremy D. Miller
@jeremydmiller
gotcha
Mark Warpool
@CodingGorilla
But at least I know I'm on the right track, I was scratching my head because I agreed with you in that I expected it to run the messages with some degree of parallelism.
Would you be opposed to a PR that defaults that to -1, which means "Unbounded" or more like "bounded only by system resources"?
Or maybe just an extension method called .Unbounded() so that it doesn't change the default behavior (maybe until a 2.0 release)
Jeremy D. Miller
@jeremydmiller
Sure. I’ll switch to Jasper soon. Or make it “number of cores” or something.
Mark Warpool
@CodingGorilla
@jeremydmiller If I need to defer handling a message for some reason, can I do something like:
public async Task<object> Handle(MyMessage message)
{
   if(this.Defer)
      return new Envelope(message).ScheduleAt(later);

   ... do normal handling ...

   return null;
}
Jeremy D. Miller
@jeremydmiller
yes
Mark Warpool
@CodingGorilla
cool, ty
Mark Warpool
@CodingGorilla
@jeremydmiller So I have an Azure App service, which can scale out instances to handle load. It uses jasper to queue up "events" to handle as data is inserted. We had a huge import run for a couple of days and what I noticed is that the jasper_incoming_messages has over 7500 messages queued up and just sitting there, they don't seem to be clearing out at all. I stopped and started the app, and the owner_id seemed to change, so it seems like Jasper knows the messages are there. Any ideas/thoughts? (All messages are set to publish locally, no transports)
Jeremy D. Miller
@jeremydmiller
Maybe database locks so nothing can take ownership? You should see worlds of logging about messages being recovered and processed, so anything coming through logging?
Possibly a poison pill if you set everything to single threaded?
Mark Warpool
@CodingGorilla
I don't see anything in the logs, but it's a production environment so only warnings and errors would appear. The max concurrency is - 1.
Jeremy D. Miller
@jeremydmiller
No exceptions whatsoever? And is there any way it's a poison pill problem, because there's no command timeout enforced by Jasper itself
Mark Warpool
@CodingGorilla
I don't think so, none of them should actually even do any work except a simple database load, because the data is old. And no, no exceptions are being logged.
Jeremy D. Miller
@jeremydmiller
When you restart, do you see the owner_id getting set to zero, then nothing? Or some of them being reset to something else?
Mark Warpool
@CodingGorilla
I see some of them being reset to something else, although after a long period I saw the "something else" id drop from a count of 200 to 199, which seems to lend some credence to your "poison pill" idea. I've fired up a local instance in the debugger and connected to production, but it doesn't seem to be attempting to recover any of the incoming messages. With the local queue set to durable, should it?
Jeremy D. Miller
@jeremydmiller
You just need the message durability turned on, but it's also been 2+ years since I've been into that code much
Mark Warpool
@CodingGorilla
ok, I'll keep digging around
Mark Warpool
@CodingGorilla
being a postgres database, it looks like the serialized message is stored binary, do you know if there's a way to see the messages?
Jeremy D. Miller
@jeremydmiller
No, but that sounds like a good idea for Jasper vNext :(
It's using some really old RhinoQueues format for writing headers and the message body in one byte stream.