Intermittent 502 EOF Gorouter errors for Spring Apps
Article ID: 298104
Updated On:
Products
Issue/Introduction
A race condition can occur between the Gorouter and a Spring backend application when keep-alives are enabled between the two servers. This race condition results in a 502 response code for the request and there is logs associated with the failed request that read "EOF" in the Gorouter stderr and stdout logs.
When keep-alives is enabled in Gorouter, the Gorouter keeps a connection alive between itself and a backend application for 90 second after the request. If an additional request comes in to the backend application instance within 90 seconds, then the Gorouter reuses that connection. For more information, refer to Gorouter Back End Keep-Alive Connection.
Note: The default keep-alive timeout setting for Tomcat is 60 seconds.
The race condition occurs when Gorouter attempts to reuse a backend connection that the Tomcat server has already started closing.
In most cases, the connection is closed successfully before Gorouter attempts to reuse it. However, requests that take more time such as POST or PUT requests, can provide just enough additional processing requirements to allow this race condition to potentially show itself.
The window for this race condition to occur is small, but it is there. For this to occur, two requests spaced 60 seconds apart would have to ingress through the same Gorouter destined for the same backend application container for this to occur.
Note: The second request in this situation is a POST or PUT request.
Environment
Resolution
Workaround
The current workaround is to increase the Tomcat connection idle timeout to a value greater than 90 seconds.
Spring Boot > v2.5 apps
For apps using Spring Boot greater than v2.5, the Tomcat connection idle timeout can be configured with the environment variable SERVER_TOMCAT_KEEP_ALIVE_TIMEOUT. Spring Boot < v2.5 apps
For apps Spring Boot less than version 2.5, a WebServerFactoryCustomizer can be used to configure this timeout programmatically.
Add this function:
public class TomcatWebServerFactoryCustomizer implements WebServerFactoryCustomizer<ConfigurableTomcatWebServerFactory> {
@Override
public void customize(ConfigurableTomcatWebServerFactory factory) {
factory.addConnectorCustomizers((connector) -> {
ProtocolHandler handler = connector.getProtocolHandler();
if (handler instanceof AbstractProtocol) {
final AbstractProtocol<?> protocol = (AbstractProtocol<?>) handler;
protocol.setKeepAliveTimeout(120000);
}
});
}
}
Alternatively, the environment variable SERVER_TOMCAT_CONNECTION_TIMEOUT may be used. It is not the same as the keep alive timeout, but should remedy the situation.
War/Tomcat apps
For War/Tomcat apps, use the Java Buildpack's external configuration option to customize the Tomcat config. Documentation for that feature is External Tomcat Configuration.
You will need to override the `server.xml` file, copy `server.xml ` and modify this line:
<Connector port='${http.port}' bindOnInit='false' connectionTimeout='20000'/>
Change the line to look like this:
<Connector port='${http.port}' bindOnInit='false' connectionTimeout='20000' keepAliveTimeout='120000'/>
There may also be other application servers or buildpacks that could potentially face this issue. If you suspect that you are hitting this issue, obtain the app logs, Gorouter logs, and contact Tanzu Support.
Feedback
