Tanzu Gorouter shows "nf_conntrack: nf_conntrack: table full, dropping packet" error in syslog

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Gorouter is getting frequent 502 and 504 errors.

gorouter.stdout.log

504 Gateway Timeout: Failed to validate Route Service Signature: route service request expired"

Syslog contains entry - 

/var/log/syslog -
2026-03-16T16:00:31.905333+00:00 ######## kernel: message repeated 9 times: [ nf_conntrack: nf_conntrack: table full, dropping packet]

 

Resolution

For immediate relief to this issue, it's recommended to horizontally scale gorouters.

The "nf_conntrack: table full, dropping packet" error means the Linux kernel's connection tracking table has reached its maximum capacity (nf_conntrack_max), causing new network packets to be dropped.

These are errors are often caused by spike in traffic / connections to gorouters .

It's recommended to monitor and alert on gorouter KPI metrics as outlined in document: https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/elastic-application-runtime/6-0/eart/monitoring-kpi.html

• gorouter.file_descriptors
• gorouter.backend_exhausted_conns
• gorouter.total_requests
• gorouter.latency
• Gorouter VM CPU Utilization

These metrics can help preemtively identify the connection exhaustion / need to scale gorouters. The following steps should be taken: 

1. Identify which app(s) are requesting excessive connections and resolve the impacting issues with these apps. This can be done by analyzing requests in gorouter access.log's.
2. If the mitigation steps have not already been taken, do so.
3. Consider adding more Gorouter VM resources to increase the number of available file descriptors and connections.