Noticing leaks in release of byte buffers

[gaurav8628]

Describe the bug
We are running an gateway application (has underlying Spring Cloud Gateway v4.0.6) that currently supports Spring Boot 3.1.x. Recently we have started to see issues of memory leaks during the handling of the responses in the filters of SCG that is impacting our application's traffic. We did try to narrow it down by disabling our custom filters and run with default SCG filters and we could still see leaks occurring when SCG is processing the response. (a partial snippet added below).

{"instant":{"epochSecond":1715881307,"nanoOfSecond":613266084},"thread":"egress-h2c-epoll-4","level":"ERROR","loggerName":"io.netty.util.ResourceLeakDetector","message":"LEAK:
ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more
information.\nRecent access records:
\n#1:\n\tio.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)\n\tio.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)\n\treactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:296)\n\treactor.netty.channel.FluxReceive.request(FluxReceive.java:131)\n\treactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164)\n\treactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:138)\n\treactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164)\n\treactor.core.publisher.Operators$BaseFluxToMonoOperator.request(Operators.java:2041)

The above log is printed when we include the jvm argument -Dio.netty.leakDetection.level=paranoid

Springboot Version: 3.1.5
Spring Cloud Gateway Version: 4.0.6
Netty Version: 4.1.100
Jetty Version: 11.0.17

This can be easily reproduced when we have RetryGatewayFilter is enabled.

[spencergibb]

Please try with the latest versions as there have been a number of fixes made in this area

[gaurav8628]

We tried with 4.0.9, we still see the same issue.
We will build a sample app and share it.

[spencergibb]

The latest version is 4.1.3

[gaurav8628]

We cannot accept 4.1.3 as yet coz as per the dependency of SCG it needs Spring Boot 3.2.x and our applications are not ready with SB 3.2.x as it has many changes.
We are currently on SB 3.1.x.

[gaurav8628]

Hi,
I am attaching the applications that I build to reproduce the issues.
demoMemoryLeak1 application have the webclient for sending the request running on port 8080
demo2 is the SCG application running on port 8089
stub-app2 is the stub application the on port 8087
Start all three application and send the curl request attached below
while true; do curl --http2-prior-knowledge --request GET 'http://localhost:8080/instances' --header 'x-http2-scheme: http' -H "Content-Type: application/json"; done
demoMemoryLeak1 2.zip
demo 2.zip
stub-app 2.zip

[gaurav8628]

I am also attaching the results of my test run for the simulation of this issue. As per my assessment the issue is arising because the SCG is not able to handle the DATA packet which is having the length as zero. I am attaching the screenshot of tcpdump along with tcpdump file and the logs which shows that Data Leak is happening.
![Screenshot 2024-05-21 at 12 02 20 PM](https:
Archive.zip
//github.com/spring-cloud/spring-cloud-gateway/assets/60240577/552250df-48d6-4fc6-a18e-b78122091401)

[gaurav8628]

The communication is on http2 please apply this filter to see the traffic for demo2 application.
tcp.port==8089 && http2

[gaurav8628]

Hi Folks, Can you please give a update regarding until when the fix will be delivered for this.