Standby Cells Inactive and Unresponsive Due to Datastore Capacity Exhaustion

Products

• VMware Cloud Director

Issue/Introduction

In a VMware Cloud Director (VCD) multi-cell environment, one or more standby cells become inactive and fail to respond to SSH or console input.

Symptoms include:

• Standby cells listed as "Inactive" or "Unresponsive" in the VCD Provider UI.

• SSH connections to the standby cells time out.

• The VM console for the standby cell is frozen and does not accept keyboard/mouse input.

• The underlying ESXi host may report storage latency or permanent device loss if the datastore is 100% full.

Environment

• 10.x

Cause

The datastore hosting the standby cell virtual machines has exhausted all available disk space, causing the virtual machines to freeze or suspend. 

Resolution

 

1. Identify the affected standby cell virtual machines in vCenter Server.

2. Power off the affected standby cell VMs.

3. Migrate the standby cell VMs to a datastore with sufficient free space using Storage vMotion or Cold Migration.

4. Power on the standby cell VMs.

5. Once the OS is booted, verify the database replication status. If the cells remain out of sync, follow the manual database node recovery procedures.

6. For detailed steps on re-integrating a standby node that has fallen out of sync, refer to Broadcom KB 1 of 2 downstream nodes not attached.