File descriptor leak causes VCD cells to crash in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director VCD cells stop responding or crash randomly.
• File descriptor (FD) count on the VCD Java process grows steadily until it reaches the system limit (e.g., 65k).
• /opt/vmware/vcloud-director/logs/vcloud-container-debug.log contains errors such as:
  • java.io.IOException: Too many open files.

Environment

VMware Cloud Director 10.6.x

Cause

The issue is caused by the Apache HttpClient implementation, where, under certain race conditions, the close call successfully closes the socket, but the underlying file descriptor is not released.

This could occur when API extensibility is used and VCD acts as a proxy where VCD cell initiates outbound connections.

Resolution

A fix is targeted for a future patch release, which will be applicable to VCD 10.6.1 and 10.6.2.

Workaround:

To mitigate the file descriptor growth until a patch is applied:

1. Block the following API calls at the AVI Load Balancer level for these specific endpoints:
   • POST /ext-ui/tenant/####/ariaops/ui/bridge/getChargebackVMs
   • GET /ext-ui/tenant/####/ariaops/ui/heartbeat.action
   • POST /ext-ui/tenant/####/ariaops/ui/metricChart.action

If disabling the API calls is not a feasible solution.

1. Perform a periodic reboot of vCloud Director cells to clear the accumulated file descriptors.

Additional Information

To be updated on the fix status, subscribe to this article by following the steps in Knowledge Base Article Subscription.