spring-cloud-gcp/Lobby

Artem Bilan

@artembilan

It might sound for you fundamental, but as you see for all these years only yo came to us asking about this feature from channel adapter implementation

Therefore feels like not so fundamental :smile:

What is fundamental that we provide a general batching solution with an Aggregator EIP implementation

Volkan Yazıcı

@vy

When people start to experience performance problems due to subscriber speed, they tend to make the mistake of thinking it is the queueing medium they need to replace, i.e., Pub/Sub. Though the problem actually the interface its APIs expose.

Artem Bilan

@artembilan

I leave a possible answer for your last comment to Google team members: I know little about that Pub/Sub API

Volkan Yazıcı

@vy

Pub/Sub wire-API (i.e., Protobuf API) is okay. It is the official driver that sucks, IMHO.

Mike Eltsufin

@meltsufin

You might be interested in this: spring-cloud/spring-cloud-gcp#2023
We never merged it though. Maybe we should prioritize it if it's something you're looking for.

Volkan Yazıcı

@vy

In case you're interested in, you can read the historical account of reactor-pubsub.

Artem Bilan

@artembilan

I see our very Sergei Egorov as contributor to your project :smile:

Volkan Yazıcı

@vy

He provided extensive reviews on the initial version, which I am very thankful of.

Mike Eltsufin

@meltsufin

We've discussed this with your coworker @mzeijen and he submitted some improvements to Spring Cloud GCP reactive Pub/Sub support.

Volkan Yazıcı

@vy

My only regret about the current state of reactor-pubsub is its reliance on HTTP 1.2, rather than HTTP 2. But this is due to HTTP 2 client support in reactor-netty. Once it is there, I will migrate from JSON-over-HTTP1 to Protobuf-over-HTTP2.

@meltsufin Right, I am aware of it.

Mike Eltsufin

@meltsufin

For batching, what do you think about the PR I referenced?

Volkan Yazıcı

@vy

@meltsufin I've skimmed through the changes and it feels like quite an improvement to the current state of things. +1

Let me share a discussion from the company's development channel:

Mike Eltsufin

@meltsufin

Feel free to give that PR a try, and if it works for you, let us know and we'll merge into master.

Mike Eltsufin

@meltsufin

@artembilan Does this look familiar: "Neither @ContextConfiguration nor @ContextHierarchy found for test"?

Never mind. I don't think that's relevant to our build failures.

Jagadeesh Manne

@mannejkumar

i need some help. Can any one tell me how to write integration tests for FirestoreReactiveRepository? for mongo db we have embedded mongodb but for firestore i didnt see any embedded db available.

Daniel Zou

@dzou

@mannejkumar Sounds like you want Firestore emulator support; basically lets you run your Firestore code against a firestore emulator which should be the equivalent to Mongo's embedded DB. We're in the progress of adding support for this here: spring-cloud/spring-cloud-gcp#2244 However, still blocked on a response from the Firestore team.

I think we might just push this through if users are interested

Prakash A

@aponmyway_twitter

Is there any development for GCP Cloud Endpoints

Mike Eltsufin

@meltsufin

What kind of integration with Cloud Endpoints do you have in mind?

We don't have anything currenlty.

Prakash A

@aponmyway_twitter

Fine

Daniel Zou

@dzou

@artembilan - Hey Artem, does Spring Integration have some built-in support for dead letter queues? Pub/sub recently added support for dead letter queues and we were wondering what extension points we could implement to make this feature easier to use in Spring Integration and Spring Cloud Stream

Artem Bilan

@artembilan

Well, Spring Integration is a light-weight messaging framework with some abstractions to adapt target protocols to/from

And since we are not broker, but library we don't hold a state

therefore it is incorrect to ask about in-memory DLQ

This is indeed a target protocol feature like JMS, AMQP or as you have just said Pub/Sub

Therefore it has to be implemented in those channel adapters

But generally speaking there is always that errorChannel abstraction which could be used for publishing into the mentioned DLQ.

In fact we even have that ErrorMessageStrategy to let to build a specific error message which could be used to produce some target protocol even for a DLQ.

Does it make sense, @dzou ?

Daniel Zou

@dzou

Thanks @artembilan. Ohh interesting... Yes that makes sense. Just trying to understand more about the function of this error channel - So right now the way this feature works is that Pub/Sub will automatically forward messages to the DLQ if the message was not-acknowledged 5 times (can be configured). This means user does not have to write an error channel to publish messages to the DLQ if this is done by Pub/Sub automatically right? Perhaps they only need to configure an error channel to read from the DLQ?

Artem Bilan

@artembilan

Well, you indeed can implement that as a fallback logic in the inbound channel adapter - sending to DLQ natively

But I don't think an errorChannel should be some how involved here if you don't want to let end-user to do something else before publishing elsewhere

Well, the more I think what you suggest the more it looks like: or native DLQ or errorChannel

with the later to be treated as a regular consumer which could publish to Pub/Sub or not

Daniel Zou

@dzou

Hmm I guess the concept of error Channel is confusing me a little now

Artem Bilan

@artembilan

OK. let me show some code then:

Daniel Zou

@dzou

Ah ok sounds good

Artem Bilan

@artembilan

protected void sendMessage(Message<?> messageArg) {
        Message<?> message = messageArg;
        if (message == null) {
            throw new MessagingException("cannot send a null message");
        }
        message = trackMessageIfAny(message);
        try {
            this.messagingTemplate.send(getRequiredOutputChannel(), message);
        }
        catch (RuntimeException ex) {
            if (!sendErrorMessageIfNecessary(message, ex)) {
                throw ex;
            }
        }
    }

See that sendErrorMessageIfNecessary()?

Daniel Zou

@dzou

Yes

Artem Bilan

@artembilan

So, if we fail to produce a message to the outputChannel, we try to send it to the errorChannel

if we haven't sent, we rethrow to the native consumer

Daniel Zou

@dzou

Ah ok I see.

Where communities thrive

People

Repo info

Activity