NSX-T Host Transport Node Out of Band Upgrade through VUM/vLCM resulting in Install_Failed status

Products

• VMware NSX

Issue/Introduction

• Upgrade-Coordinator and Edges upgrade completed from NSX Manager UI
• On the Host Transport-node upgrade step, upgrade is performed Out of Band (OOB) by attaching VUM baseline image of target version in vLCM
• After this, Host component Upgrade status went into PAUSED state and Host TN Status in UI shows as Install_Failed. When Resolve option is clicked, we see below error:

Node has invalid version 4.2.2.1.##### of software nsx-monitoring. Node has invalid version 4.2.2.1.##### of software nsx-vdpi. Node has invalid version 4.2.2.1.##### of software nsx-shared-libs. Node has invalid version 4.2.2.1.#####........... .....

Environment

VMware NSX

Cause

• Issue is caused as recommended upgrade steps were not performed while upgrading a Host TNs OOB
• In case of OOB upgrade, the Fabric Module on the NSX manager is not updated with the new/target version and it still lists the older/source version as the current version
• When the VIBs are updated on the Host TN, Fabric module running on the NSX Manager sees a software mismatch and tries to install the older/source version on the Host TNs
• As the Host TNs are already on the newer/target version and downgrade is not supported, this results in host status being in Install_Failed with above error
• We can see Fabric Module version using API as below:

GET https://nsx-manager/api/v1/fabric/modules

Example: Consider upgrade Target version is 4.2.2.1.##### and the Source/Old version is 3.2.4.0.##### , below snippet from the API response needs to be considered:

{

            "fabric_module_name": "hostprep",

            "current_version": "3.2.4.0.#####" ------> Should point to 4.2.2.1.#####

}

Resolution

Resolution:

User should either perform Host upgrade from UC or perform Stage-upgrade in case of OOB Host upgrade as mentioned in the 4.2 Upgrade Guide (Step 2) - Upgrade Hosts Manually

 

Workaround:

• To workaround this issue, please follow one of the steps below: 

• • From NSX UI : 
    Navigate to the Host Upgrade tab and Click Stage and proceed after staging is complete
     4.2 Upgrade guide (Step 2) - Upgrade Hosts Manually

OR

• • Use Below API to update the Fabric Module:

POST https://<nsx-mgr>/api/v1/upgrade/plan?action=stage-upgrade&component_type=HOST

• Confirm if the 'hostprep' section of the Fabric Module is updated to the Target version by running the following API call on the Orchestrator Manager node:

 GET https://nsx-manager/api/v1/fabric/modules

 

If you are unsure whether this article applies to your situation, need help executing any of the steps, or require assistance with validation, please open a case with the Broadcom Technical Support Team using the knowledge base link below.

Creating and managing Broadcom support cases