NSX Upgrade the upgrade fails with an error "The Upgrade has failed. Appliance OS is of old version"

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T upgrade fails with an error displayed on the manager UI
  Error : “The Upgrade has failed. Appliance OS is of old version, however UI will not be available Please contact GSS to retry the upgrade.”
• This happens only if Intelligence node is present and backup-restore is done
• You will see progress status as below

Node-nsx01b> get upgrade progress-status
****************************************************************************
Node Upgrade has been started. Please do not make any changes, until
the upgrade operation is complete. Run "get upgrade progress-status"
to show the progress of last upgrade step.
****************************************************************************

Upgrade info:
From-version: 3.0.1.0.0.16404476
To-version: 3.0.2.0.0.16887203

Upgrade steps:
download_os [2020-10-28 15:42:08 - 2020-10-28 15:42:43] SUCCESS
shutdown_manager [2020-10-28 15:42:46 - 2020-10-28 15:44:41] FAILED
    Status: cloud-init status command returned output active.

    Stdout: Starting shutdown script

    Stderr: Traceback (most recent call last):
  File "/image/VMware-NSX-unified-appliance-3.0.2.0.0.16887203/scripts/shutdown_manager.py", line 404, in <module>
    main(args)
  File "/image/VMware-NSX-unified-appliance-3.0.2.0.0.16887203/scripts/shutdown_manager.py", line 380, in main
    ret = disable_and_shutdown(options, output)
  File "/image/VMware-NSX-unified-appliance-3.0.2.0.0.16887203/scripts/shutdown_manager.py", line 266, in disable_and_shutdown
    current_version = get_current_version()
  File "/image/VMware-NSX-unified-appliance-3.0.2.0.0.16887203/scripts/shutdown_manager.py", line 36, in get_current_version
    with open("/repository/current_version", "r") as fd:
FileNotFoundError: [Errno 2] No such file or directory: '/repository/current_version'

    Troubleshooting: Upgrade has failed. Appliance OS is of old version, however, UI will not be available. Please contact GSS to retry the upgrade

Note:The preceding log excerpts are only examples.Date, time and environmental variables may vary depending on your environment

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center 2.5.x

Cause

REPO_SYNC had failed on all the nodes in the cluster that led to the failure in updating the file of Current Version, eventually leading to the failure of MP node upgrades.

Resolution

This issue is resolved in NSX-T versions 2.5.3 and 3.0.3

Workaround:
Upload NSX Intelligence OVA file before performing restore. For more information, refer to VMware Documentation

Additional Information

Impact/Risks:
The upgrade would have started, and it will fail with manager and control plane cluster in a DOWN state.