NSX Manager 4.2.4 Upgrade Hangs at 70% and MP upgrade failed due to failed to start Playbook

Products

• VMware NSX

Issue/Introduction

When upgrading to VMware NSX 4.2.4, one or more NSX Manager nodes exhibit the following behavior:

• Migrated the NSX appliances (Edges and Managers) to a supported hardware generation.
• Power cycle all Edge nodes one at a time within an Edge cluster however no power cycle was done for the associated NSX Manager nodes.
• The NSX Manager upgrade progress in the UI hangs at 70%
• NSX Manager upgrade failed with error "[MPP] Node upgrade failed : MP upgrade failed due to failed to start Playbook."
• Below logs are seen under /var/log/upgrade-coordinator/upgrade-coordinator.log

[Timestamp]  INFO pool-126-thread-1 UpgradeServiceImpl 879798 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Component type = MP , Upgrade Sta
tus = FAILED, Progress Percentage = 8 , Progress Message = Creating local backup
Creating local backup on [NSX-MGR-IP] node
Creating local backup on [NSX-MGR-IP] node
Creating local backup on [NSX-MGR-IP] node
Successfully created local backup
Updating version allowlist for controller cluster
Putting node [NSX-MGR-UUID] in maintenance mode.
[[NSX-MGR-IP]] Preparing MP node [NSX-MGR] for upgrade
[[NSX-MGR-IP]] Upgrade bundle verified on management node [NSX-MGR]
[[NSX-MGR-IP]] Starting playbook on [NSX-MGR]
[[NSX-MGR-IP]] Playbook started on [NSX-MGR]
[[NSX-MGR-IP]] MP upgrade step shutdown_manager in progress.
[[NSX-MGR-IP]] MP upgrade step install_os in progress.
[[NSX-MGR-IP]] MP upgrade step migrate_manager_config in progress.
[[NSX-MGR-IP]] MP upgrade step reboot in progress.
[MPP] Node upgrade failed : MP upgrade failed due to failed to start Playbook..

errors=[{"moduleName":"upgrade-coordinator","errorCode":30062,"errorMessage":"Unexpected error while upgrading upgrade unit: [MPP] Node upgrade failed : MP upgrade failed due to failed to start Playbook.."}], metaData={manager_ip=[NSX-MGR-IP], manager_ipv6=, UU_TYPE=NODE, manager_ipv4=[NSX-MGR-IP]}

• The Management Plane becomes unreachable (no Ping, SSH, or Web UI) until a manual power cycle is performed.

 

Environment

VMware NSX 4.2.4

Cause

VMware NSX 4.2.4 unified appliances have a new dependency on the RDSEED CPU instruction set for random number generation and entropy during the boot sequence. 

This issue occurs in environments where:

1.  The physical host CPU is of an older generation (e.g., pre-Broadwell for Intel) that does not support RDSEED.

2.  The vSphere Cluster is configured with a legacy Enhanced vMotion Capability (EVC) baseline (e.g., Haswell or earlier), which masks the RDSEED instruction from the guest OS even if the physical hardware supports it.

Without sufficient entropy from RDSEED, the NSX services (Proton) fail to initialize within the allotted time, causing the upgrade playbook to time out.

Resolution

To resolve this issue, Power cycle all NSX Manager nodes one at a time and continue with the upgrade.

This is known issue as described on NSX-T 4.2.4 Manager/Edge appliances fail to install or upgrade due to missing RDSEED support on unsupported CPU or incompatible EVC configurations

Note: Make sure to remediate all NSX appliances (Edges and Managers) before retrying and continuing with the upgrade.