NSX-T Host Transport Node Out of Band Upgrade through VLCM resulting in Install_Failed status

Products

• VMware NSX

Issue/Introduction

• Upgraded NSX-T Upgrade-Coordinator and Edges from NSX-T UI
• On the Host Transport-node upgrade step, upgrade is performed Out of Band (OOB) by attaching vum baseline image of target version in VLCM
• After this, Host component Upgrade status went into PAUSED state and Host TN Status in UI shows as Install_Failed. When Resolve option is clicked, we see below error:

Node has invalid version 4.2.2.1.0-7.0.xxxx of software nsx-monitoring. Node has invalid version 4.2.2.1.0-7.0.xxx of software nsx-vdpi. Node has invalid version 4.2.2.1.0-7.0.xxx of software nsx-shared-libs. Node has invalid version 4.2.2.1.0-7.0.xxx........... .....

Environment

VMware NSX

Cause

• Issue is caused as proper/supported upgrade steps were not performed while upgrading a Host TN OOB
• In case of OOB upgrade, the Fabric Module on the NSX manager is not updated with the New/Target version and it still lists the older/source version as the current version
• When the VIBs are updated on the Host TN, Fabric module running on the NSX manager sees a Software Mismatch and tries to install the older/source version on the Host TNs
• As the Host TNs are already on the newer/target version and downgrade is not supported, this results in host status being in install_failed with above error
• We can see Fabric Module version using API as below:

GET https://nsx-manager/api/v1/fabric/modules

Example: Consider upgrade Target version is 4.2.2.1.XXXX and the Source/Old version is 3.2.4.0.0.xxxxx , below snippet from the API response needs to be considered:

{

            "fabric_module_name": "hostprep",

            "current_version": "3.2.4.0.0.xxxxx" ------> Should point to 4.2.2.1.XXXX

Resolution

Workaround:

• To workaround this issue, please follow one of the steps below:

1. From NSX UI : 
   Navigate to the Host Upgrade tab and Click Stage and proceed after staging is complete
    4.2 Upgrade guide (Step 2) - Upgrade Hosts Manually
2. Use Below API to update the Fabric Module:

POST https://<nsx-mgr>/api/v1/upgrade/plan?action=stage-upgrade&component_type=HOST

Confirm if the 'hostprep' section of the Fabric Module is updated to the Target version using API 'GET https://nsx-manager/api/v1/fabric/modules'

Resolution:

User should either perform host upgrade from UC or performed staging-upgrade in case of OOB as mentioned in the 4.2 Upgrade guide (Step 2) - Upgrade Hosts Manually

If you are unsure whether this article applies to your situation, need help executing any of the steps, or require assistance with validation, please open a case with the Broadcom Technical Support Team using the knowledge base link below.

Creating and managing Broadcom support cases