reactor/reactor-netty

Reactive Network Driver based on Netty and Reactor

Tripplesixty

@Tripplesixty

Question about logging. Why is that in all the http2 pipelines we do not add the metricsRecorder (HttpClientMetricsRecorder) to the end of the pipeline. I see we do this for http1, but not for h2. For reference I don't see it being added here: https://github.com/reactor/reactor-netty/blob/baff0517529357a4e45b79d900114971519e1a68/reactor-netty-http/src/main/java/reactor/netty/http/client/HttpClientConfig.java#L506

note: I do see my custom metrics class being invoked for methods that inherit from ChannelMetricsRecorder, but not from HttpClientMetricsRecorder. Thanks!

2 replies

Piotr Borczyk

@pborczyk

hi, I have a problem with a writeWith() method of ReactiveHttpOutput interface, I pass a flux of DataBuffers to it obtained from Spring's WebClient. However no data is being sent despite the fact that the passed flux of databuffers is not empty. The browser does not recieve the http body. I've debuged deep into reactor and ended up in FluxReceive class. The databuffer flux is never being consumed. It only happens when the flux is of webclient origin it works with byte array backed fluxes.

3 replies

Glennos

@glennosss

Topic: Websockets - detecting client close (or more broadly, any close of the websocket).

The websocket reactive stream can be closed - which in turn closes websocket (WebsocketSubscriber.onComplete() calls WebsocketServerOperations.sendCloseNow() if the websocket is still active).

However when the client closes the websocket the reactive stream isn't closed (I would have assumed WebsocketSubscriber.onSubscribe() should store the subscription to a field, then call 'subscription.cancel()' when the client closes the websocket).

The issue I've got is I want to maintain a pool of active websockets to broadcast messages to and I need to be able to remove websockets from this pool when they disconnect. The API doesn't especially allow for this, what I would have assumed, was a very standard usecase.

1 reply

Zane XiaoYe

@Zane-XY

hi @violetagg
can we use Webflux's Webclient to send web requests in QUIC protocol?

3 replies

Mohammad Salem

@mhmdsalem1993

I’ve got this error when using netty HttpClient with Spring WebClient
java.util.NoSuchElementException: Context does not contain key: callereventloop
Any idea what might be the cause?

12 replies

Alex Pavlovich

@alexp1917

Hi all, I am looking for information on using netty http clients (or spring abstractions) with 1) arbitrary http methods 2) streaming response bodies (as Flux<byte[]>, Flux<ByteBuf>, or Flux<DataBuffer> etc...).

4 replies

Snofty

@snofty

Hi Team, we have an application with spring boot 2.6.4 and webflux. currently facing a java.lang.OutOfMemoryError: Direct buffer memory

Original Stack Trace: Caused by: java.lang.OutOfMemoryError: Direct buffer memory     at java.nio.Bits.reserveMemory(Unknown Source) ~[?:?]     at java.nio.DirectByteBuffer.<init>(Unknown Source) ~[?:?]     at java.nio.ByteBuffer.allocateDirect(Unknown Source) ~[?:?]     at io.netty.buffer.UnpooledDirectByteBuf.allocateDirect(UnpooledDirectByteBuf.java:104) ~[netty-buffer-4.1.74.Final.jar:4.1.74.Final]     at io.netty.buffer.UnpooledDirectByteBuf.capacity(UnpooledDirectByteBuf.java:156) ~[netty-buffer-4.1.74.Final.jar:4.1.74.Final]     at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:305) ~[netty-buffer-4.1.74.Final.jar:4.1.74.Final]     at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280) ~[netty-buffer-4.1.74.Final.jar:4.1.74.Final]     at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1103) ~[netty-buffer-4.1.74.Final.jar:4.1.74.Final]     at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:99) ~[netty-codec-4.1.74.Final.jar:4.1.74.Final]     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:277) ~[netty-codec-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.74.Final.jar:4.1.74.Final]     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.74.Final.jar:4.1.74.Final]     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.74.Final.jar:4.1.74.Final]     at java.lang.Thread.run(Unknown Source)

I have tried -Dio.netty.leakDetection.level=paranoid
but didn't give any leak details. can you please guide or help me to investigate

3 replies

Mohammad Salem

@mhmdsalem1993

Currently im investigating a memory leak with netty and Spring webflux
The logs doesnt contain the connection id so that i can know which connection has the memry leak
I’ve the following enabled : io.netty.leakDetection.targetRecords=40, -Dio.netty.leakDetection.level=paranoid, logging.level.reactor.netty.channel.FluxReceive=debug
but this is the log i get:

LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)

I expect something with the connection id like this:

LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [id: 0x0f9d0570, L:/10.36.23.97:48846 - R:www.googleapis.com/108.177.111.95:443] Buffered ByteBufHolder in Inbound Flux Queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)

any ideas if im doing something wrong so that i can get the connection id?

13 replies

NathanNorth

@NathanNorth

Trying to pin down an issue causing an intermittent

io.netty.handler.codec.EncoderException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1

The only place in the entire project where I deal with bytebufs is in this method (httpRequest is a ResponseReceiver):

public Mono<String> getRequest() {
        return httpRequest.responseSingle(((response, byteBufMono) -> {
            if(response.status().code() / 100 == 2) {
                return byteBufMono.asString(StandardCharsets.UTF_8);
            }
            else {
                return byteBufMono.asString(StandardCharsets.UTF_8)
                        .switchIfEmpty(Mono.just(""))
                        .flatMap(data -> Mono.error(FailureStrategies.makeWebException(response, data)));
            }
        }));

The only thing I could possibly think of is that I call getRequest() multiple times with the same ResponseReciever if I get a 500 or whatever, but that retry system works most of the time so idk why that would be the issue.

1 reply

Franck Brisbart

@franck.brisbart_gitlab

Hi all, I'm trying to enable brotli compression, but after digging in the code (reactor.netty.http.server.HttpServerOperations), it seems it's not possible. Anyone can confirm and/or any clue on how doing that ?

20 replies

Bill Koch

@billkoch

Hi All - I saw that reactor-netty-1.0.15 introduced two server metrics that I'd like to expose in a spring-boot-webflux application (reactor.netty.http.server.connections.active and reactor.netty.http.server.connections.total. I'd like to avoid turning on the other reactor.netty.http.server.* metrics because a number of them are tagged with URIs which will generate high cardinality in my metrics, and I already have spring-boot generating metrics that include templated URIs for me.

Is it possible to selectively register specific metrics that will be exposed? To avoid the high cardinality, I registered a NettyServerCustomizer bean that registers all the metrics with a URI tag as "":

public class NettyServerMetricsCustomizer implements NettyServerCustomizer {
  @Override
  public HttpServer apply(HttpServer httpServer) {
    return httpServer.metrics(true, s -> "");
  }
}

But this seems less than ideal. I appreciate any help/suggestions!

2 replies

lalifk

@lalifk

Hello,
I'm using a reactor netty server and would like to count how many SSL handshakes took place. Is there a way to do so using metrics for example? This is with reactor-netty version 1.0.15

5 replies

Claudio Tasso

@taxone

Hi,
I'm trying to figure out the cause of an issue which is also present in stackoveflow. This is what I get in my logs:

"2022-03-22 11:53:38.061 DEBUG 1 --- [     parallel-1] o.s.w.r.f.client.ExchangeFunctions       : [6fc37ed8] HTTP POST https://myserver.info/oauth/v2/token\n",
"2022-03-22 11:53:48.048 DEBUG 1 --- [or-http-epoll-4] o.s.w.r.f.client.ExchangeFunctions       : [6fc37ed8] Cancel signal (to close connection)\n"

I have not set any timeout configuration, so it seems that some kind of timeout occurs (always after 10 seconds). There aren't exceptions in the logs, it just seems that the connection is closed. I tried to reproduce this behavior simulating timeout or connection closing on the server, but in such cases I always get an error and an exception in the logs.

Have you any idea about what could happen to the connection or how can I reproduce this error?
Thank you very much.

Anil Gursel

@anilgursel

Hi all, Is there a way to access the Netty Pipeline from ContextAwareHttpMetricsRecorder methods? What I am trying to do is to add TLS protocol and cipher names as dimensions to TLS handshake metrics in ContextAwareHttpMetricsRecorder. It has access to ContextView but, not sure how I can put TLS related stuff to it. SslHandler in Netty pipeline has this details, but not sure how to access to it.

3 replies

Snofty

@snofty

Hi Team, with spring boot 2.6.4 trying to print webclient response in logs (both success or failure cases) and trying below option.

var httpClient = HttpClient.create(connectionProvider)
                .wiretap(getClass().getSimpleName() + " Request/Response : ", LogLevel.INFO, AdvancedByteBufFormat.TEXTUAL);

is it fine to have above code on production or does it has performance impact? please

6 replies

Zane XiaoYe

@Zane-XY

@violetagg
I’m using reactor-netty httpclient with a load balancer. The LB has a default connection time out of 180s. I want to simulate a "connection reset by peer error", reset by the LB specifically. I assume if I set the client with

maxIdleTime(Duration.ofSeconds(360))           
.maxLifeTime(Duration.ofSeconds(360))

it should reproduce this error a lot, but I can hardly see any. I got around 1 or 2 over 24 hours, I not even sure this was related to the settings.
What kind of setting can help me reproduce this error? and how does the maxIdleTime affect if the client exceeds the configuration?

2 replies

Zane XiaoYe

@Zane-XY

Hi, is there any way that I can enable the io.netty.handler.ssl.SslMasterKeyHandler#newWireSharkSslMasterKeyHandler to export my SSL master key log for debugging purposes? I’m using the Webflux WebClient.

7 replies

Olivier

@ogorostis

Hi everyone,
I just would like to understand better the rationale for blocking retries once some data has been sent at
https://github.com/reactor/reactor-netty/blob/main/reactor-netty-http/src/main/java/reactor/netty/http/client/HttpClientConnect.java#L360
Is it simply : once some data has been sent (whether it be by body or by headers), who knows what the serving end has done, hence let's not retry because better safe than sorry ?
Thanks.
Olivier

4 replies

Bennie Sun

@BennieSun

ReactorNettyWebSocketClient - Connecting to ws://192.168.0.101:11024/websocket/ws?accessToken=456
2022-04-10 21:43:06,408 [reactor-http-nio-3] ERROR o.s.w.s.a.HttpWebHandlerAdapter - [61cd3240-9] Error [java.lang.UnsupportedOperationException] for HTTP GET "/api/websocket/ws?accessToken=456", but ServerHttpResponse already committed (200 OK)

spring-boot : 2.3.12 RELEASE
reactor-netty : 0.9.20 RELEASE

1 reply

why already committed cause the connection to be closed

Mohammad Reza Sabouri Karkhaneh

@sabouri

Hello! I have the following code, which simply posts to httpbin. I expected it to re-use the HTTP connection, but from the logs I get the impression that the connection is immediately closed after each request, and re-opened for the next requests.
Is there some misconfiguration, or am I misunderstanding the logs?
Thanks!

        ConnectionProvider provider = ConnectionProvider.builder("custom")
                .maxConnections(16)
                .maxIdleTime(Duration.ofSeconds(100))
                .maxLifeTime(Duration.ofSeconds(100))
                .build();

        HttpClient httpClient = HttpClient.create(provider)
                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 5000)
                .responseTimeout(Duration.ofMillis(5000))
                .doOnConnected(conn ->
                        conn.addHandlerLast(new ReadTimeoutHandler(5000, TimeUnit.MILLISECONDS))
                                .addHandlerLast(new WriteTimeoutHandler(5000, TimeUnit.MILLISECONDS)))
                .wiretap("reactor.netty.http.client.HttpClient",
                        LogLevel.DEBUG, AdvancedByteBufFormat.TEXTUAL);

        httpClient.warmup()
                .block();

        while (true) {
            httpClient.post()
                    .uri("https://httpbin.org/post")
                    .send(ByteBufFlux.fromString(Mono.just("hello")))
                    .response()
                    .block();
            try {
                Thread.sleep(10_000);
            } catch (InterruptedException e) {
                e.printStackTrace();
                break;
            }
        }

5 replies

Anders Clausen

@AndersClausen

Hi all. Does anybody know how to set Server Name Indication (SNI) in reactor-netty? Thank you

2 replies

Richard Kollcaku

@ricardkollcaku

Hi
Im trying to do some benchmarking with spring boot using http1 and http2
and i see i weird behaviour.
SPRING BOOT 2.6.6
http2 is bit slower than http1 looks like is not taking advantage of single tcp connection with multiplexing.

My Server:
I have enabled http2 in properties :
server.http2.enabled= true
and in using curl

curl -I --http2 http://localhost:8080/sms-request/test

i get:

HTTP/1.1 101 Switching Protocols
connection: upgrade
upgrade: h2c

HTTP/2 200 
content-type: text/plain;charset=UTF-8
content-length: 10

My client is another spring boot 2.6.6
and looks like

 WebClient webClient = WebClient.create().mutate()
       .clientConnector(
            new ReactorClientHttpConnector(
                HttpClient.create().protocol(HttpProtocol.H2C)))
        .baseUrl("http://localhost:8080/sms-request/test").build();

    Flux.range(1, 100000000)
        .concatMap(integer -> webClient.get().retrieve().bodyToMono(String.class))
      .subscribe()
    ;

Using default protocol i can process up to 23,959 requests per second
with H2C protocol i can barely reach 16.000request per second

6 replies

vsv00001

@vsv00001

Hi,
I have a spring cloud gateway application that has a filter that does a post to a third party service. It is working when i send a body
as Mono.empty() but is failing when i send a simple json request body .
spring cloud gateway version : 2.2.6.RELEASE
"errorMessage": "java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf"

"exception":"io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf
 io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104)
 io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346) 
 io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
 io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
 io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) 
 io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) 
 reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:317) 
 reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:419) 
 reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:219) org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:90) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)  org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:90) reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96)
 reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344) 
 reactor.core.publisher.FluxContextStart$ContextStartSubscriber.request(FluxContextStart.java:125) org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:76) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:76) reactor.netty.channel.MonoSendMany$SendManyInner.onSubscribe(MonoSendMany.java:250) org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:69) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:69) reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onSubscribe(FluxContextStart.java:90)\n\tat 
 reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) 
 reactor.core.publisher.Flux.subscribe(Flux.java:8325) 
 reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102) 
 reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153) 
 reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
 reactor.core.publisher.Mono.subscribe(Mono.java:4213) 
 reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:331)
 reactor.core.publisher.MonoSource.subscribe(MonoSource.java:65) 
 reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) 
 reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)

this is the truncated exception message i am seeing. Can some one please help me in figuring out why this could be happening.

Thanks

Ahmed Abdul-Hamid

@ahmed-abdulhamid

I have a bit of code (below) that depends on reactor-netty:0.9.20.RELEASE but no longer compiles after upgrading to 1.0.17. I saw a reference to this snippet as a potential fix in an earlier thread but, as a casual maintainer, it's not obvious to me if that covers (2) below (i.e. setting the channel Class).

Any help is greatly appreciated. Thanks in advance.

fun webServerFactory(): NettyReactiveWebServerFactory {
       val eventLoopGroup = /* omitted for brevity */
       val serverChannelClass = /* omitted for brevity */

        val eventLoopCustomizer = NettyServerCustomizer { server ->
            server.tcpConfiguration { tcpServer ->
                tcpServer.bootstrap { serverBootstrap ->
                    serverBootstrap
                        .group(eventLoopGroup)           // <-- (1) custom EventLoopGroup
                        .channel(serverChannelClass)     // <-- (2) custom channel Class 
                }
            }
        }

        return NettyReactiveWebServerFactory().also { factory ->
            factory.serverCustomizers = listOf(eventLoopCustomizer)
        }
}

5 replies

HOA PHAN

@s50600822

Hi can anyone point me to the class that build the metric "reactor.netty.http.client.data.sent" type DistributionSummary that mentioned on https://projectreactor.io/docs/netty/release/reference/index.html#_metrics_6 ?
Not reactor.netty.http.client.data.sent.time, I can see reactor.netty.http.client.data.sent.time in reactor.netty.http.client.HttpClientMeters I just can't find the DistributionSummary one

2 replies

I guess MicrometerChannelMetricsRecorder ?

dileep

@dileepkmandapam_twitter

HI @violetagg . I have a reactive TCP server running fine in AWS. But the server is logging the following WARNINGS randomly (10 times in day and 4 such instances in a month). Unfortunately I cannot enable DEBUG mode in prod.

2022-04-23T16:51:52,737 [reactor-tcp-epoll-3] WARN  r.n.c.FluxReceive {} - An exception has been observed post termination, use DEBUG level to see the full stack: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Error finalising cipher data: mac check in GCM failed

Could you help me in finding the root cause. Thanks in advance.

Paolo Venturi

@pv3nturi

Hi, i'm working on connection pools and ConnectionProviders for a http client connections. I'm struggling with "hot-reload" of working providers: i can't update settings of a working provider (ex max connections, etc), the only way seems to create a brand new one. However if i dispose the current provider in favor of the new one, all metrics stop working and the new provider seems not to register any data. There is any way to update a ConnectionProvider? May be a bug thats metrics stop working for providers configured when the application is running? (for any reference, here is the project code https://github.com/diennea/carapaceproxy/blob/902a0aeda7571911567a7a8a7120572da139afcf/carapace-server/src/main/java/org/carapaceproxy/core/ProxyRequestsManager.java#L483)

1 reply

Claudio Tasso

@taxone

My application is based on Spring Cloud Gateway. I'd like to log the Sleuth trace identifier, but the trace context is always empty. This is a snapshot of the stack trace:

It seems that during connection creation the trace context is not created.

11 replies

Vivek Ashokan

@vivekitis

hello. i had a doubt related to spring-webflux DefaultWebClient class.
Is this the right group to ask? Thank you.

Zane XiaoYe

@Zane-XY

Hi @violetagg
my WebClient connects to a LB directly, and LB has a max idle timeout of 180s. I experimented that setting maxConnection to 1, and maxLifeTIme, maxIdleTime longer than 180s (360s), even after 180s, the connection is still useable. I’m expecting to get a connection reset error or something. Why is that? If the LB sends a FIN to close the connection, will the WebClient clean that connection from the pool and start a new connection?

5 replies

Vaibhav Bansal

@vaibhavflip

hi @violetagg
we are using spring webflux, with latest spring boot version 2.6.6, and have been facing issue with memory leaks. We use webclient for our external http calls and spring reactive redis driver for interacting with redis. After enabling detection level to paranoid, we can see following stacktrace added in the thread. please help advise what else we can do to pinpoint the issue.

4 replies

Vaibhav Bansal

@vaibhavflip

hey all, does anyone have idea about how to add _reactor.netty.channel.LeakDetection in application.yml ?

7 replies

ChristianLMI

@ChristianLMI

Hi, I'm currently trying to make sure our services based on reactor run as solid as possible. Today we had an issue after configuring metrics. Apparently (unfortunately our logs where not sufficient to be absolutely sure) we had requests - using Springs WebClient - fail after a NullPointer Exception was observed. As we had NullPointerExceptions also in recording metrics for the incoming call in the MetricsWebFilter (due to an error a tag was created without key and value), I believe this was the same issue while adding the same tag in the client metrics. Other than in MetricsWebFilter the recording that I believe happens in ContextAwareHttpMetricsRecorder is not surrounded by a try catch block. This causes the production workload to fail if metrics cannot be recorded. I believe this is not great.
I'd try to create a PR to improve the behavior, if this is s.th. that seems desirable. Any thoughts?

13 replies

Dhairyashil

@dhairyashil

Hi, I am getting below exception while calling apis after sometime.

java.nio.channels.ClosedChannelException: null
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1064) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]

Any thoughts?

1 reply

Abhishek Pandey

@AbhiPandeyIU_twitter

This message was deleted

2 replies

Sushant Parkhi

@sushantparkhi

Hi, I am currently trying to make ECDSA related ciphers to work with TLS 1.2 in Spring Cloud Gateway (Spring Boot Parent 2.6.7 and Spring Cloud 2021.0.2). Here's the snippet of WebServerFactoryCustomizer

    @Bean
    public WebServerFactoryCustomizer<NettyReactiveWebServerFactory> customizer() {
        return factory -> factory.addServerCustomizers(httpServer -> httpServer.secure(sslContextSpec -> {
            try {
                Ssl ssl = factory.getSsl();
                KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
                char[] keyStorePassword = ssl.getKeyStorePassword().toCharArray();
                keyStore.load(resourceLoader.getResource(ssl.getKeyStore()).getInputStream(), keyStorePassword);
                KeyManagerFactory keyManagerFactory = OpenSslCachingX509KeyManagerFactory
                        .getInstance(KeyManagerFactory.getDefaultAlgorithm());
                keyManagerFactory.init(keyStore, keyStorePassword);
                Http11SslContextSpec http11SslContextSpec = Http11SslContextSpec.forServer(keyManagerFactory)
                        .configure(sslContextBuilder -> {
                            sslContextBuilder.sslProvider(SslProvider.OPENSSL);
                            sslContextBuilder.ciphers(Arrays.asList(ssl.getCiphers()));
                            sslContextBuilder.protocols(ssl.getEnabledProtocols());
                            sslContextBuilder.trustManager(InsecureTrustManagerFactory.INSTANCE);
                            sslContextBuilder.clientAuth(ClientAuth.REQUIRE);
                        });
                sslContextSpec.sslContext(http11SslContextSpec)
                        .handlerConfigurator(sslHandler -> {
                            sslHandler.setCloseNotifyReadTimeout(18000, TimeUnit.MILLISECONDS);
                            sslHandler.setHandshakeTimeout(19000, TimeUnit.MILLISECONDS);
                            SSLParameters sslParameters = sslHandler.engine().getSSLParameters();
                            sslParameters.setUseCipherSuitesOrder(false);
                            sslHandler.engine().setSSLParameters(sslParameters);
                        });
            } catch (UnrecoverableKeyException | IOException | CertificateException | KeyStoreException |
                     NoSuchAlgorithmException e) {
                throw new RuntimeException(e);
            }
        }));
    }

But when I try to connect using openssl s_client with ECDHE-ECDSA-AES128-GCM-SHA256 cipher the server returns an error with no shared ciphers, but I do have it in the configuration as

server.ssl.ciphers=TLS_RSA_WITH_AES_128_GCM_SHA256,\
  TLS_RSA_WITH_AES_256_GCM_SHA384, \
  TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,\
  TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,\
  TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
server.ssl.enabled-protocols=TLSv1.2

This behavior was observed when I upgraded versions from Spring Boot 2.3.3.RELEASE and Spring Cloud Hoxton.SR7. Any advice/suggestions would be of great help on fixing or correctly configuring it.

sarowar

@eipi1

Hi,
Is there any way to limit the maximum number of connections for reactor netty?
I've got a web flux REST API that's accepting more requests than it can process. So I'd like to set a maximum limit to concurrent active connections or/and accept queue size.
Thanks in advance.

1 reply

Hamado Dene

@hamadodene

Hi, we need some info about httpClient usage. It's possibile to use the same httpClient for multiple requests? In addition, can we setup custom headers for every request? I mean if something like this is possibile:

HttpClient client =  HttpClient.create(connectionProvider)
                    .host(endpointHost)
                    .port(endpointPort)
                    ...

//Firt requests
String response1 = client.headers(h -> h.add(request.getRequestHeaders().copy())
                    .request("GET")
                    .uri("http://example1.com/")
                    .send(request.getRequestData())
                    ..

//Second request                    
String response1 = client.headers(h -> h.add(request.getRequestHeaders().copy())
                    .request("GET")
                    .uri("http://example2.com/")
                    .send(request.getRequestData())
                    ..

Thanks in advance.

6 replies

Muhammad Ilyas

@ilyas2016

Hello everone, I am using webflux with spring boot version 2.6.7 when i add the blockhound it notified me that netty is internally calling some blocking function i.e FileInputStream when i use webclient, how could i prevent this blocking operation for now?

1 reply

ChristianLMI

@ChristianLMI

Hi, I'm trying to build reactor-netty in order to create a PR. It does build on the CLI using gradlew build. However in my IDE (Spring Tools Suite) the project is missing the shaded dependencies from reactor.pool. Any ideas how I can provide them, or even better change configuration for the IDE to find them?

4 replies

Claudio Tasso

@taxone

Hi, I'm facing some connection issues in a Kubernetes cluster where Netty acts as a server.
The clients are traditional Spring MVC applications which use a standard RestTemplate.
Sometimes, when the clients try to connect to an application based on Reactor Netty, they obtain two kinds of exceptions:

java.net.SocketException: Operation timed out (Read failed)
java.net.SocketException: Connection reset

There are other cases where the connection is successful, but the client connection arrives to the Netty server after a lof of time (30 seconds or more).
The Netty server is NOT under heavy load and no errors are present in the logs.
Moreover, this seems to happen when the Netty server has been idle for several hours, i.e. the incoming connection is the first one after a long period of inactivity.
Has anyone experienced the same issue?
Does any configuration parameter (server side) exsist in Netty which could mitigate this issue?

9 replies

moremind

@moremind

why remove ConnectionProvider#elastic in netty-core?I can't find this reason.

1 reply

Sam Lightfoot

@samlighty_twitter

Hi - I noticed the default value for maxActiveStreams is Integer.MAX_VALUE for a H2 connection. In my head this could lead to a single connection being overloaded when perhaps other connections with much fewer active streams would be more suitable to process a given request. Is there a reason for Reactor Netty not setting a more suitable default value? Microsoft seem to think 100 is a sensible value.
Are there other mechanisms in HTTP2 connection selecting that I am missing that would prevent a single connection being overloaded?

4 replies

STC

@siddhantthapak

Hi, I am using webclient to call services. I am using java-springboot. While making call to service using webclient, I am getting SSLHandShakeTimeOutException: handshake timed out affter 10000ms
Then I set the handShakeTimeMillis to 18000. Now I am getting I/O exception: An existing connection was forcibly closed by the remote host.
I tried setting up ConnectionProvider.fixed("test", 64). It did not work. Can somebody please help me to resolve this issue. I would be grateful!!

I am using reactor-netty of version 0.9.17.RELEASE
SslContext sslContext = SslContextBuilder.forClient().trustmanager(InsecureTrustmanagerFactory.INSTANCE).build();
ClientHttpConnector httpConnector = new ReactorClientHttpConnector(HttpClient.create().secure(t-> t.sslContext(sslContext).handshakeTimeoutMillis(18000)));

And I have used httpConnector to build webClient.

Hamado Dene

@hamadodene

Hi, Sometimes we are getting in production enviroment the Exception: Error: reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response .. We are not able to reproduce it in Dev enviroment. Debugging with the logs we see that occured when we send data to Tomcat, and before receive all response. Do you have any idea how I might approach this problem?
The problem rarely happens and it is difficult for us to set up a tcp dump in production.

2 replies

Where communities thrive

People

Repo info

Activity