JasperFx/marten

Postgresql as a Document Database and Event Store for .Net Applications

Oskar Dudycz

@oskardudycz

Maybe it’s worth bringing back

radeanurazvan

@radeanurazvan

Hi. I'm wondering if Project methods part of the same EventProjection are not meant to be dependent on each other? We have Project(SomethingCreated) and Project(SomethingElseHappened) that is using the view created in the first Project, but the second Project is always failing because it cannot find the view that should have been created in the first Project? We're running into this after we created a new Projection and it is trying to catch up

Oskar Dudycz

@oskardudycz

Could you provide a sample code? Best if it contains config, projection and how you’re storing events

radeanurazvan

@radeanurazvan

Will provide one when I get back home, but basically Project(SomethingCreated) does ops.Store(new View()) and Project(SomethingElseHappened) does var view = ops.Query<View>().Where(filter).SingleAsync() and it always throws and nothing gets in the db table, assumingly because all Project are executed in a single transaction

Oskar Dudycz

@oskardudycz

The operations are batched, so records might not be there

Jeremy D. Miller

@jeremydmiller

@radeanurazvan @oskardudycz The sessions used by the event projections are all lightweight (for performance), meaning that there’s no identity map between usages. The event projection was never meant to be used that way. You’re better off using one of the aggregation projections if you’re applying multiple events to the same document

And to Oskar’s point, the updates are batched — again for performance.

radeanurazvan

@radeanurazvan

In scenarios where for one event we need to update multiple instances of the same projection, but not each instance is updated with the same behavior, which type of projection is recommended to be used?

Yücel Kıvanç

@yclkvnc

Hello everyone, I'm evaluating Marten for a future project where I work. I've read @oskardudycz 's article: "https://event-driven.io/en/ordering_in_postgres_outbox/" and I'm wondering if that technique can be used in Async Daemon to poll event table?

Oskar Dudycz

@oskardudycz

Hi, Async Daemon for now is using a different technique to maintain the ordering guarantee: gap detection.

It's more sophisticated as it's trying to detecct if the gap occurred because of rollback or ongoing transactions.

20 replies

What I described is used partially in v3.x. We were discussing that with @jeremydmiller and maybe some of that we'll also include into Marten. But for now, you should be safe while using Async Daemon

Jeremy D. Miller

@jeremydmiller

@yclkvnc At this point, I would say that what the async daemon already does is the most robust, possible approach and that the current “high water mark” detection scheme is battle tested through user feedback at this point.

Oskar Dudycz

@oskardudycz

just to add, other solutions, e.g. Java Axon are using also gap detection to keep the order guarantee, so it's also the established pattern

John Tesh

@johnteshnats_twitter

we are using the following to fetch events for a stream, however we'd like to ignore some events if possible in the same call. is this something this method can handle?
var martenEvents = await _documentSession.Events
.FetchStreamAsync(stream_id)
.ConfigureAwait(false);

Oskar Dudycz

@oskardudycz

FetchStream cannot do that, but you can use QueryAllRawEvents and filter with Linq

1 reply

Jeremy D. Miller

@jeremydmiller

@oskardudycz & @mysticmind, I’ll go active tomorrow reviewing PRs for Marten. I think two more weeks will hopefully get me through Wolverine 1.0

18 replies

sudhakshinagirish

@sudhakshinagirish

We are using marten with Aggregate Projections and wondering if there was a hook to the Apply() method to be able to set some common fields instead of having to do it on each Apply
e.g.
If all Trip events derived from a common class say TripEvents with fields
DateTime ActionedOn
string ActionedBy

and Trip has
LastUpdatedBy
LastUpdatedOn

And Apply currently is defined as
public void Apply(TripEnded e, Trip trip)
{
trip.Active = false;
trip.EndedOn = e.Day;
SetCommonFields() // sets LastUpdatedBy and LastUpdatedOn from the ActionedBy and ActionedOn
}

Then I would like to Set the common fields in a generic manner with a hook such as OnApply(TripEvents events) so all events will always set these metadata fields and is not left to the discretion of individual developers to do so.

I checked out the IDocumentListener , but that appear to happen much later in the cycle during OnSaveChanges. Is there something available earlier?

Jeremy D. Miller

@jeremydmiller

I believe that you can by having an Apply() that takes in some kind of common base type or interface. Might check the aggregation document for that

5 replies

Oskar Dudycz

@oskardudycz

I merged .NET 7 PRs

@jeremydmiller I created issues for you:

The first one should be probably easy for you.

The second may be environmental (only on CI).
Both already existed on the main branch.

Jeremy D. Miller

@jeremydmiller

Woot! Got it, probably not today, but at least this week

Oskar Dudycz

@oskardudycz

:+1: :slight_smile:

Toby Knott

@codeConcussion

We're finally upgrading from Marten 3 to 5. Where did the Store.Schema.ToPatch() method go? Is there another way to generate the patch sql now?

Jeremy D. Miller

@jeremydmiller

I assume since you're coming from 3 there's no database per tenant? Try IDocumentStore.Storage.Default.*****

I don't have the code open, but the docs point a little bit at that: https://martendb.io/schema/migrations.html

And see the CLI usage too

Toby Knott

@codeConcussion

Thanks, @jeremydmiller.

For posterity (or myself a year from now), the code went from this...

var patch = documentStore.Schema.ToPatch(false, true);
var ddl = patch.UpdateDDL;

to this...

var migration = await documentStore.Storage.Database.CreateMigrationAsync();
var migrator = new PostgresqlMigrator()
{
    Formatting = SqlFormatting.Pretty,
    UpsertRights = SecurityRights.Definer,
    TableCreation = CreationStyle.CreateIfNotExists,
    IsTransactional = true
};

using var writer = new StringWriter();
migration.WriteAllUpdates(writer, migrator, AutoCreate.CreateOrUpdate);
var ddl = writer.ToString();

Giuseppe Patanè

@GiuseppePatane

hello, maybe it's a stupid question, but is it possible to implement a projection that writes to an external system like Elastic search ?
I've been trying martin for a couple of hours to see if it's right for my use case.

13 replies

DanCymonz

@DanCymonz

anyway i can use QueryAllRawEvents while taking advantage of the watermark logic to prevent reading wip events?

Jeremy D. Miller

@jeremydmiller

Not directly, no. Not something that's ever come up before. You could make an initial query to find the high water mark, then use that value in a Linq clause I suppose. Why though?

31 replies

Alexander Zeitler

@AlexZeitler

I tried something like this to work around the unsupported is operator error:

var documents = session.Query<Document>()
  .Where(
    d => d.Tags.Any(
      t => t.MatchesSql(
        "data ->> '$type' = ?",
        typeof(InvoiceTag).MartenTypeName()
      )
    )
  )
  .ToArray();

Works fine.

Now I wanted to make it more generic and ended up this way:

public static bool Is<T>(this object obj)
{
  return obj.MatchesSql(
    "data ->> '$type' = ?",
    typeof(T).MartenTypeName()
  );
}

var documents = session.Query<Document>()
  .Where(d => d.Tags.Any(t => t.Is<InvoiceTag>()))
  .ToArray();

However, I get this exception when using the generic method:

System.NotSupportedException: Marten does not (yet) support Linq queries using the MartenQuerying.Tests.ArrayTypeQueries.TypeExtensions.Is() method

Any chance to get my little wrapper working?

Jeremy D. Miller

@jeremydmiller

You might actually be in luck: https://martendb.io/documents/querying/linq/extending.html

Check out the method call parser business. Remember within the Linq parsing that it never actually executes your extension method, it's interpreting the code

Alexander Zeitler

@AlexZeitler

Thanks, sounds like a plan!

Jeremy D. Miller

@jeremydmiller

:thumbsup:

Alexander Zeitler

@AlexZeitler

That's the first shot which solved it:

public static class CustomExtensions
{
  public static bool IsType<T>(this object target)
  {
    return target.GetType() is T;
  }
}

public class IsType: IMethodCallParser
{
  public bool Matches(MethodCallExpression expression)
  {
    return expression.Method.Name == nameof(CustomExtensions.IsType);
  }

  public ISqlFragment Parse(IFieldMapping mapping, ISerializer serializer, MethodCallExpression expression)
  {
    var martenTypeName = expression.Method.GetGenericArguments()
      .First()
      .MartenTypeName();
    return new WhereFragment("data ->> '$type' = ?", martenTypeName);
  }
}

I really love the fact with Marten there's never a dead end (so far), even something is not supported oob.

Michael Lehto

@Leh2

Any known issues with indexing of noda time types? Getting the error functions in index expression must be marked IMMUTABLE when trying to index an instant. Added failing tests here JasperFx/marten#2413

Jeremy D. Miller

@jeremydmiller

There is now:). Date types have been the scourge of Marten from the get go.

Michael Lehto

@Leh2

Great :) Do you want an issue or is it enough with the pull request?

teddy-albina

@TeddyAlbina

Hello everyone, i'm currently searching for a transactional CAP compliant database to replace Dynamodb, so my question is can Marten support complex documents ? We have a a lot of objects that contains dictionaries of objects that have other dictionaries etc. Thanks you all.

Jeremy D. Miller

@jeremydmiller

So Marten will have the same transactional model as Postgresql itself, so you're already better off on that. Marten can handle any document that can be bi-directionally serialized by either System.Text.Json or Newtonsoft.JSON, which is basically anything

Oskar Dudycz

@oskardudycz

See e.g. https://martendb.io/documents/hierarchies.html#document-hierarchies

Vinko Vrsalovic

@koide

Hello - I want to test rebuilding projections and I'm not really sure about how to proceed. Can I access the started daemon via AddAsyncDaemon() somehow, or that is only for async projection building and I should use the store.BuildProjectionDaemon() approach to do rebuilding?

3 replies

Jeremy D. Miller

@jeremydmiller

@mysticmind @oskardudycz If it's alright w/ y'all, I'd like to push a new Marten 6 alpha today with the latest, consolidated dependencies. So, Baseline -> JasperFx.Core, BaselineTypeDependency -> JasperFx.TypeDiscover, LamarCodeGeneration -> JasperFx.CodeGeneration etc. No changes in functionality.

I've got Weasel, Oakton, and Lamar (which doesn't impact Marten) updated so far

Oskar Dudycz

@oskardudycz

:+1:

Where communities thrive

People

Repo info

Activity