NSX Upgrade the upgrade fails with an error "The Upgrade has failed. Appliance OS is of old version"

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T upgrade fails with an error displayed on the manager UI
  Error : “The Upgrade has failed. Appliance OS is of old version, however UI will not be available Please contact GSS to retry the upgrade.”
• This happens only if Intelligence node is present and backup-restore is done
• You will see progress status as below

Node-nsx01b> get upgrade progress-status
****************************************************************************
Node Upgrade has been started. Please do not make any changes, until
the upgrade operation is complete. Run "get upgrade progress-status"
to show the progress of last upgrade step.
****************************************************************************

Upgrade info:
From-version: 3.0.1.0.0.16404476
To-version: 3.0.2.0.0.16887203

Upgrade steps:
download_os [2020-10-28 15:42:08 - 2020-10-28 15:42:43] SUCCESS
shutdown_manager [2020-10-28 15:42:46 - 2020-10-28 15:44:41] FAILED
    Status: cloud-init status command returned output active.

    Stdout: Starting shutdown script

    Stderr: Traceback (most recent call last):
  File "/image/VMware-NSX-unified-appliance-3.0.2.0.0.16887203/scripts/shutdown_manager.py", line 404, in <module>
    main(args)
  File "/image/VMware-NSX-unified-appliance-3.0.2.0.0.16887203/scripts/shutdown_manager.py", line 380, in main
    ret = disable_and_shutdown(options, output)
  File "/image/VMware-NSX-unified-appliance-3.0.2.0.0.16887203/scripts/shutdown_manager.py", line 266, in disable_and_shutdown
    current_version = get_current_version()
  File "/image/VMware-NSX-unified-appliance-3.0.2.0.0.16887203/scripts/shutdown_manager.py", line 36, in get_current_version
    with open("/repository/current_version", "r") as fd:
FileNotFoundError: [Errno 2] No such file or directory: '/repository/current_version'

    Troubleshooting: Upgrade has failed. Appliance OS is of old version, however, UI will not be available. Please contact GSS to retry the upgrade

Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center 2.5.x

Cause

REPO_SYNC had failed on all the nodes in the cluster that led to the failure in updating the file of Current_Version, eventually leading to the failure of MP node upgrades.

Resolution

This issue is resolved in NSX-T versions 2.5.3 and 3.0.3

Workaround:
Upload NSX Intelligence OVA file before performing restore. For more information refer to VMware Documentation

Additional Information

Impact/Risks:
The upgrade would have  started  and it will fail with  manager and control plane cluster in a DOWN state.