VMware Cloud Foundation (vCF) skip level upgrades to 3.10.2 may be affected by NSX direct upgrade limitation

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:
vCF Skip level upgrades may be affected by an issue with direct upgrades from NSX-V versions 6.4.2, 6.4.3, 6.4.4, and 6.4.5 to NSX-V 6.4.10

You will see an error similar to the following in the SDDC-Manager logs, /var/log/vmware/vcf/lcm/lcm.log:

2021-06-08T23:34:34.094+0000 WARN [87eald2549a151f4,829a,upgradeId=ebe05f27-7411-49cc-a4b4-58f0fc372061,resourceType=NSX_MANAGER,resource1d=d5789fb5-4a13-4978-a9f2-4711fd96dcee,bundleElementId=32c2ebcd-bb4a-4967-9267-2aS7943a4b98] [c.v.e.s.l.c.v.nsx.NsxManagerClient,ThreadPoolTaskExecutor-10] 
Bad request querying upgrade info: {"errors":[{"details":"Upgrade target version 6.4.10.17626462 is same or less than the current version 6.4.5.","errorCode":151015."rootCauseString":null,"moduleName":"vsm-appliance-mgmt","errorData":null}]} 
2021-06-08T23:34:34.094+0000 ERROR [87eald2549a151f4,829a,upgradeId=ebe05f27-7411-49cc-a4b4-58f0fc372061,resourceType=NSX_MANAGER,resourceId=d57891b5-4a13-4978-a9f2-47111d86dcee,bundleElementId=32c2ebcd-bb4a-4967-9267-2a57943a4b98] [c.v.e.s.l.p.i.nsx.NsxPrimitiveImpl,ThreadPooliaskExecutor-10] Unhandled exception during NSX component upgrade: java.lang.NullPointerException: null
    at com.vmware.evo.sddc.lcm.client.vmware.nsx.NsxManagerClient.startManagerUpgrade(NsxManagerClient.java:463) 
    at com.vmware.evo.sddc.lcm.primitive.impl.nsx.services.NsxManagerUpgradeService.upgradeManager(NsxManagerUpgr adeService.java:157) 
    at com.vmware.evo.sddc.lcm.primitive.impl.nsx.services.NsxManagerUpgradeService.doUpgradeStage(NsxManagerUpgr adeService.java:319) 
    at com.vmware.evo.sddc.lcm.primative.impl.nsx.NsxPrimitiveImml.runUpgrade(NsxPrimativeImpl.java:301) 
    at com.vmware.evo.sddc.lcm.primitive.impl.nsx.NsxPrimitiveImpl.startOrResumeUpgrade(NsxPrimativeImpl.java:168)

Environment

VMware Cloud Foundation 3.7.x
VMware Cloud Foundation 3.8.x
VMware Cloud Foundation 3.5.x
VMware Cloud Foundation 3.9.x

Resolution

Currently, there is no resolution.

Workaround:

Customers planning to upgrade from any of the impacted releases can follow one of the two options below:

Option-1: Beanshell Script (Recommended)

1. Download the attached signed_bsh_UpgradeUtilsFix_withVersionCheck.encoded file.
2. Run the following REST API POST call using either method a. or method b. as described below.  Choose one method or the other before moving on to step 3.                                                              You can choose to use a REST API Client like POSTMAN on NSX Manager. Alternatively, you can use a curl command as mentioned below in case you are executing this from a Linux machine.

a. First Method Option: POSTMAN

URL: https://NSXMGR_IP/api/1.0/services/debug/script
(Replace the NSXMGR_IP in the above URL with the actual IP address of NSXMGR)

Authentication: Basic authentication (Username: admin)

Headers: content-type - application/xml

Body: copy contents of the attached file signed_bsh_UpgradeUtilsFix_withVersionCheck.encoded.

Sample screenshot:




OR

b. Second Method Option: Using curl command for Linux:

curl --verbose --noproxy '*' -u admin:'default' -H "Accept: application/xml" -H "Content-Type: application/xml" -k -X POST --data @/tmp/signed_bsh_UpgradeUtilsFix.encoded 

https://NSXMGR_IP/api/1.0/services/debug/script

Expected Response: 200

3. When the BeanShell patch is applied, restart the appliance management service with the below api. Refer to the screenshot:

POST https://NSXMGR_IP/api/1.0/appliance-management/components/component/APPMGMT/restart 

4. Next, Retry the upgrade from the SDDC Manager UI 

Additional Information

Impact/Risks:
Deployments of vCF 3.9 and prior will experience failures with skip level upgrades directly to 3.10.2.