netty/netty

Netty project - an event-driven asynchronous network application framework

Martin Furmanski

@martinfurmanski

Ah right, I had a vague recollection there was a blocker for that

That would be cool

Norman Maurer

@normanmaurer

let me think about it a bit

Martin Furmanski

@martinfurmanski

Thanks!

Enrico Olivelli

@eolivelli

So if I have hundreds of threads that are allocating ByteBufs (using PooledByteBufAllocator) I will have a lot of churn, but the global memory counter will be correct

Norman Maurer

@normanmaurer

Yes I would say so

Enrico Olivelli

@eolivelli

Thank you @normanmaurer ! I will be back with news after heap dump analysis

Norman Maurer

@normanmaurer

jmnght

@jmnght

@normanmaurer @violetagg seeing below memory leak in production after updating reactor to 3.2.5, spring-boot to 2.1.2, Netty to 0.8.4 . Would you have any thoughts?

[03-14-19 13:32:49.141]  [reactor-http-epoll-3] ERROR  i.n.util.ResourceLeakDetector LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:171)
    io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:105)
    org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:65)
    org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39)
    org.springframework.http.codec.json.AbstractJackson2Encoder.encodeValue(AbstractJackson2Encoder.java:172)
    org.springframework.http.codec.json.AbstractJackson2Encoder.lambda$encode$0(AbstractJackson2Encoder.java:122)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107)
    reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2070)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162)
    reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:1878)
    reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:1752)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90)
    reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
    reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
    reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44)
    reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
    reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1505)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
    reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
    reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2070)
    reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:138)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onSubscribe(MonoFlatMap.java:230)
    reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:172)
    reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
    reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
    reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2070)
    reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:1878)
    reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:1752)
    reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.Mono.subscribe(Mono.java:3695)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75)
    reactor.core.publisher.Operators.complete(Operators.java:131)
    reactor.core.publisher.MonoEmpty.subscribe(MonoEmpty.java:45)
    reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44)
    reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44)
    reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
    reactor.core.publishe

Violeta Georgieva

@violetagg

@jmnght reactor/reactor-netty#613

Norman Maurer

@normanmaurer

@martinfurmanski netty/netty#8941 PTAL

@eolivelli also maybe interesting for you ^^

Enrico Olivelli

@eolivelli

@normanmaurer this is the memory counter patch netty/netty#8945 thanks.

I am working on the heap dump (but it is huge), but I think that current discussion already helped me a lot in understanding the mechanics of using Netty allocators out side Netty threads. So I think I won't find much valueable data in the dump.

Norman Maurer

@normanmaurer

@eolivelli ok cool… let me know if you have any more questions etc

Norman Maurer

@normanmaurer

@eolivelli @martinfurmanski the PR was merged

Enrico Olivelli

@eolivelli

@normanmaurer I saw. Longing now for a release :-)

Martin Furmanski

@martinfurmanski

@normanmaurer Tres cool. Also looking forward to the next release!

Lukasz

@lradziwonowicz

Hi, simple application which returns fastest result of two WebClient requests using Mono.first(). The application is failing with LEAK error:

2019-03-28 10:52:21.203 ERROR 25237 --- [or-http-epoll-4] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.

I prepared POC with full error logs: https://github.com/lradziwonowicz/spring-boot-webflux-webclient-netty-leak Should I create a bug or my approach is not correct?

Norman Maurer

@normanmaurer

@lradziwonowicz I think you should ask in the reactor / spring forums

it basically tells you release() was not called. I guess this should be done as part of the reactor / spring integration tho

Violeta Georgieva

@violetagg

@lradziwonowicz let’s discuss it here https://gitter.im/reactor/reactor You might want to try with the latest Californium-SR6 (put this property <reactor-bom.version>Californium-SR6</reactor-bom.version> in the pom.xml)

Actially from the code - you consume just the status code and do not consume the body and thus your code will leak the memory

you might want to check these two links

https://docs.spring.io/spring/docs/current/spring-framework-reference/web-reactive.html#webflux-client-exchange

bodyToMono(Void.class)

Violeta Georgieva

@violetagg

https://stackoverflow.com/questions/51312537/what-is-the-proper-way-to-make-an-api-call-via-springs-webclient-but-ignore-t/51321602#51321602

Lukasz

@lradziwonowicz

@violetagg ok, that was only an example, my original application was consuming the body, where to put Californium-SR6 inside of pom.xml, <properties>?

Violeta Georgieva

@violetagg

yep

Lukasz

@lradziwonowicz

ok, but maven did not download anything new, is this ok?

Violeta Georgieva

@violetagg

Reactor Core should be version 3.2.8 and Reactor Netty 0.8.6?

Lukasz

@lradziwonowicz

ok, I see it in dependency tree, I'll test it, thx for the links

Violeta Georgieva

@violetagg

:+1:

Lukasz

@lradziwonowicz

@violetagg cool, it is working, the LEAK is gone. I'll test it with my original application

Violeta Georgieva

@violetagg

also you might want to try then Spring Framework 5.1.6 snapshots there are some fixes related to memory leaks

Nouras Hamwi

@NourasHamwi

I am using netty with apache camel and it works fine on my local enviroment ( Mac ) but when I moved the project to Linux it is giving this error NettyProducer WARN No payload to send for exchange

Any idea what could be causing it ?

Norman Maurer

@normanmaurer

Sorry I think you will need to ask on the camel mailing list

Nouras Hamwi

@NourasHamwi

I got the issue resolved, it was a packaging proglem.

Enrico Olivelli

@eolivelli

Hi guys. I am working on Maven Surefire. We are going to rewrite the communication system between core Maven process and forked JVMs in tests, we are going to use loopback network instead of stdout/stdin. We are currently discussing about using Netty because it will make it very simple to implement the protocol and it is, you know, very efficient.
I have a concern about shading/relocating Netty bits and deploying it inside the forked JVM: this will "pollute" the environment for projects that are really using Netty.
May I have an opinion from any of you ?

Norman Maurer

@normanmaurer

not sure what to say here @eolivelli :D

I think as it is only a test dependency I would not worry

that said I think you must shade

netty is too widely used to not shade

Enrico Olivelli

@eolivelli

How would you see it to have a shaded org.apache.surefire.io.netty while running netty self tests ? Wouldn't it be a mess, looking at thread dumps for instance, or contents of ThreadLocals, log messages ?

Norman Maurer

@normanmaurer

hmm no idea

I would not be too concerned imho

as long as it is shaded

Enrico Olivelli

@eolivelli

ok, thanks for your feedback @normanmaurer

Norman Maurer

@normanmaurer

Francesco Nigro

@franz1981

Hi! I'm looking the single threaded event loops and which type of JCTools q they use...

Where communities thrive

People

Repo info

Activity