Upgrade Aborted Due to Non-Homogeneous Cluster Node Resources
Article ID: 451498
Updated On:
Products
Issue/Introduction
Upgrade to 31.2.1 and higher fails during the WaitUntilClusterReady task,with the following timeout error, indicating that the cluster is not active:
Timeout: waited 1200 sec. and the cluster is still not active.
Once the upgrade process has aborted, you can locate the timeout error by running the following command on the Controller CLI
shell
show upgrade status detail filter controller
Symptoms:
- The upgrade process times out (default 1200 seconds) during the WaitUntilClusterReady task.
- The upgrade status is marked as UPGRADE_FSM_ABORT_IN_PROGRESS.
- Logs in cluster_mgr.INFO or upgrade_events show the following error:
Cluster nodes need to be homogeneous regarding memory, CPU, disk, version, patch version, and FIPS mode
Path: /var/lib/avi/log/cluster_mgr.INFO
ERROR [cluster_quorum_manager.raise_event_and_update_local_state_key:596] ^[[31m[QUORUM] [MEMBER_PROBE] Clustify compatibility check failed for node node3.controller.local with error {'Summary': ['Cluster nodes need to be homogeneous regarding memory, CPU, disk, version, patch version, and FIPS mode.'], 'Detail': {'##...##': '[ *CPU: 8,* Memory: 33.66, \{*}Disk: 125.19{*}, Major Version: \{30.2.6 9035}, Patch Version: , Fips mode: false ]', '##...###': '[ *CPU: 6,* Memory: 33.66, \{*}Disk: 503.15{*}, Major Version: \{32.1.2 9077}, Patch Version: , Fips mode: false ]'}}^[[0m
INFO [cluster_events.savePendingEvent:71] Saved pending cluster event: /var/lib/avi/etc/pending_cluster_events/CLUSTIFY_CHECK_EVENT-7667438469370868507.ev
INFO [cluster_quorum_manager.set_local_node_state_key:228] [QUORUM] [CLUSTER_STATE_UPDATE] Setting local node state key CLUSTIFY_CHECK_FAILURE for node3.controller.local
Environment
- Version: 31.2.1 and later
Cause
The upgrade fails because at least one node in the cluster has a different resource configuration (CPU count, Memory size, or Disk capacity) compared to the other nodes. This often happens after resource hot-plugging without a subsequent VM reboot, or when a node is manually resized inconsistently.
Starting with version 31.2.1, strict node-homogeneity enforcement was introduced in the clustify service to ensure cluster stability. If the CPU, memory, or disk configurations are not identical across all nodes, the cluster will fail to form, and the upgrade will abort.
Resolution
To resolve the homogeneity conflict and proceed with the upgrade, perform the following steps:
- Verify the current resource configuration in the host_resources.conf file on all three controller nodes.
- Path: /var/lib/avi/etc/host_resources.conf
- Check the values for num_cores, mem_GB, and disk_GB. If the values are not identical then go to the next step.
cd /var/lib/avi/etc cat host_resources.conf disk_GB: 125.18648147583008 mem_GB: 33.655627776 num_cores: 8
- Ensure that the VM settings in the hypervisor (e.g., vCenter) match across all nodes for CPU, Memory, and Disk.
- Schedule a maintenance window to reboot the Controller VMs to ensure the host_resources.conf file is updated with the correct values.
- Follow the documented reboot sequence:
a. Shut down all Controller VMs, starting with the Follower nodes and ending with the Leader node.
b. Confirm that Memory, CPU, and Hard Disk settings are identical across the cluster.
c. Start the Leader Controller VM first.
d. Once the Leader is online, start the Follower Controller VMs. - Verify that the cluster is healthy with all nodes in ACTIVE Sate and the host_resources.conf values are now identical across all nodes.
- Restart the upgrade process.
Additional Information
- For detailed steps on resizing, refer to the Broadcom Tech Docs: [Upgrading Controller Size for Controller Cluster]
Feedback
