File descriptor leak causes VCD cells to crash in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director VCD cells stop responding or crash randomly.
• File descriptor (FD) count on the VCD Java process grows steadily until it reaches the system limit (e.g., 65k).
• /opt/vmware/vcloud-director/logs/vcloud-container-debug.log contains errors such as:

  java.io.IOException: Too many open files

Environment

VMware Cloud Director 10.6.x

Cause

The issue is caused by the Apache HttpClient implementation, where, under certain race conditions, the close call successfully closes the socket, but the underlying file descriptor is not released.

This could occur when API extensibility is used and VCD acts as a proxy where a VCD cell initiates outbound connections.

Resolution

Raise a support request with Broadcom to request a hot patch.

Note: Applying the hot patch will upgrade the environment to 10.6.2 first before applying the Hot Patch.

Workaround:

Option 1:

Block the following API calls at the AVI Load Balancer level for these specific endpoints:

• POST /ext-ui/tenant/####/ariaops/ui/bridge/getChargebackVMs

• GET /ext-ui/tenant/####/ariaops/ui/heartbeat.action

• POST /ext-ui/tenant/####/ariaops/ui/metricChart.action

Option 2:

• Perform a periodic reboot of vCloud Director cells to clear the accumulated file descriptors.

Additional Information

To be updated on the fix status, subscribe to this article by following the steps in Knowledge Base Article Subscription.