Gorouter VMs Failing and Microservices Timing Out Due to Kernel nf_conntrack Table Exhaustion

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Customer may observe severe network degradation and routing failures within the Tanzu Platform for Cloud Foundry (TPCF) environment. Symptoms include:

• Gorouter VM Instability: Gorouter VMs continuously flap and report a "failing" state in BOSH.
• Application Timeouts: Applications experience intermittent I/O timeouts when attempting to communicate with other internal microservices or the Cloud Controller API.
  
  

  ERR: Request error: Post "https://api.<system_domain>/v3/spaces": dial tcp X.X.X.X:443: i/o timeout

• Monit Failures: The monit process crashes intermittently on the affected VMs, yielding the following error:
  

  monit-actual: error connecting to the monit daemon

• DNS Failures: bosh-dns crashes or fails to establish connections to external DNS resolvers.

 

Environment

Tanzu Platform for Cloud Foundry (TPCF) / Elastic Application Runtime (EAR) v10.x

Cause

The root cause is the exhaustion of the Linux kernel's connection tracking table (nf_conntrack).

The nf_conntrack module tracks all active network connections traversing the Gorouter. When the number of concurrent connections exceeds the maximum limit configured in the kernel (nf_conntrack_max), the kernel begins aggressively dropping new packets.

Because packets are dropped at the kernel level:

• Microservice-to-microservice API calls time out.
• bosh-dns queries are dropped, causing resolution failures.
• Monit's internal network checks fail to reach the daemon, causing the VM health checks to fail and flap.

During troubleshooting, it was observed that the concurrent connections on the Gorouters were rapidly reaching ~55,000 to 60,000, hitting the maximum threshold allowed for a VM with 4GB of RAM.

Note: Scaling out the number of Gorouter VMs horizontally (e.g., from 6 to 9) does not adequately resolve this issue if the traffic profile inherently drives high per-node connection counts.

Resolution

Temporary Workaround:-
To immediately restore service and stop the kernel from dropping packets, you can dynamically increase the maximum connection tracking limit directly on the affected Gorouter VMs without a reboot.

• SSH into the failing Gorouter VMs.
• Run the following sysctl command to increase the threshold to 262,144:
  

  sudo sysctl -w net.netfilter.nf_conntrack_max=262144

Permanent Solution:-
The nf_conntrack_max value is automatically calculated by the Linux kernel based on the total amount of RAM allocated to the VM.

To permanently resolve this issue so that the configuration survives VM recreation and BOSH deployments:

• Navigate to Ops Manager > TAS Tile > Resource Config.
• Locate the Router (Gorouter) job.
• Scale up the VM Type to ensure the VM has at least 8GB of RAM (up from 4GB).
• Click Save and Apply Changes.

By increasing the VM memory to 8GB, the kernel will automatically calculate a much higher nf_conntrack_max limit upon boot, safely accommodating the environment's high connection volume.

Additional Information

To monitor the current connection tracking usage and limits on a Linux VM, you can run the following commands:

• Check current tracked connections: sudo sysctl net.netfilter.nf_conntrack_count
• Check the maximum limit: sudo sysctl net.netfilter.nf_conntrack_max