Cluster Stuck in “Waiting to Finish Cluster Expansion” state after adding new node

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

In Aria/VCF Operations, a cluster may get stuck in the "Waiting to finish cluster expansion" state during node addition or expansion. Additionally, if enabled, High Availability (HA) may report as Activated, degraded.

Environment

Aria Operations 8.x
VCF Operations 9.x

Cause

The issue occurs due to insufficient disk space on the /storage/db partition, which is critical for database operations. The following command can be run to verify disk space df -h


Resolution

Follow these steps to resolve the issue:

• SSH into all of the nodes as root user
• On each node, execute the following command to take the slice offline safely:

  $VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOffline --offlineReason "Offline_Reason"

• Follow the Step 2 to 14 in the KB: How to add disks to a node in Aria Operations cluster
• Log in to the Admin UI (https://<PrimaryNode_IP_FQDN>/admin) as the admin user.
• The “Finish Adding New Nodes” button will now be enabled.
  
• Click on this button to complete the cluster expansion process and the cluster will come Online.

Additional Information

Expand an Existing Installation to Add a Data Node