VMware NSX General error or java.lang.NullPointerException error during upgrade

Products

VMware NSX

Issue/Introduction

Symptoms:

A DELETE API call to upgrade-mgmt/plan might have been run prior to this issue
There maybe an error message as "uc.upgradeExecuteError: General error"
During edge upgrade, after successful precheck, there may also be another error displayed in the UI: "java.lang.NullPointerException"

In the log /var/log/upgrade-coordinator/upgrade-coordinator.log, there are log entries similar to:

INFO localhost-startStop-1 UpgradePlanServiceImpl 9971 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Persisting Plan {"upgrade_status":"NOT_STARTED","source_version":null,"target_version":null,"component_plans":{"MP":{"component_type":"MP","plan_settings":{"upgradeMethod":"SERIAL","pauseOnError":true,"pauseAfterEachGroup":false},"groups":

ERROR http-nio-127.0.0.1-7442-exec-2 UpgradeServiceImpl 745049 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30078" level="ERROR" subcomp="upgrade-coordinator"] Error while continuing the upgrade for component EDGE
...
INFO localhost-startStop-1 UpgradePlanServiceImpl 966306 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Persisting Plan {"upgrade_status":"NOT_STARTED","source_version":null,"target_version":null,"component_plans":{"MP":{"component_type":

2024-04-04T15:46:42.801Z ERROR http-nio-127.0.0.1-7442-exec-2 UpgradeServiceImpl 745049 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30078" level="ERROR" subcomp="upgrade-coordinator"] Error while continuing the upgrade for component EDGE
java.lang.NullPointerException: null

Environment

VMware NSX-T and VMware NSX 4.1.x

Cause

This issue is caused by the API call that is used to delete the upgrade plan, after the UC upgrade has been completed and the upgrade has begun.

The issue occurs when the target_version is null, in some instances the issue source_version may also be null.

Resolution

****When the upgrade has been started, the MUB has been uploaded and upgrade coordinator service has been upgraded, do not use the API call to delete the upgrade plan, this API has been removed from future releases.***

Workaround below is applicable if upgrade-coordinator.log and API call in step 2 show "source_version" and "target_version" or "target_version" as null:

On all 3 NSX-T managers, as root user, run:
# /etc/init.d/upgrade-coordinator stop
On 1 of the NSX-T managers, as root, run the following API to export the upgrade plan for review and edit:
curl -k -X GET -u admin 'https://localhost/api/v1/upgrade-mgmt/plan' -H 'Content-Type: text/plain' -H 'Accept: text/plain' > /tmp/upgrade_plan.json

Note: You do not need to use the root user, you can also use an API tool such as Postman but make sure you change the accept-encoding to "plain" or else it will show "unsupported media type" error.
Edit exported file /tmp/upgrade_plan.json, and change the null values from:
("source_version":null,"target_version":null)
to
("source_version":"<source version>","target_version":"<target version>")

For example if you're upgrading from 3.2.2.1.0.21487560 to 4.1.0.0.0.21332672, the edit should like the below:
("source_version":"3.2.2.1.0.21487560","target_version":"4.1.0.0.0.21332672")

Note: As mentioned above, it may only be the target version which is null, if so, you only need to edit the "target_version" field.
Then using the edited upgrade_plan.json file, use the following PUT API call against 1 of the NSX-T managers:
curl -k -u admin -X PUT 'https://localhost/api/v1/upgrade-mgmt/plan' -H 'Content-Type: text/plain' -H 'Accept: text/plain' -d '@/tmp/upgrade_plan.json'
Then as root user, start the service again on all 3 managers:
# /etc/init.d/upgrade-coordinator start
Proceed with the upgrade