Upgrade of NSX on ESXi host fails with VDR Failed to perform full sync

Products

• VMware NSX

Issue/Introduction

• NSX upgrade on ESXi host has status "Failed".
• NSX UI shows error: Hosts upgrade has failed, check error details to determine if manual resolution is needed and 'Retry Upgrade'.
• Log lines similar to the below are encountered in /var/log/upgrade-coordinator/upgrade-coordinator.log:

  ERROR task-executor-4-1-workitem-HOST-########-####-####-####-############ Esx60SoftwareVibCliInvokeImpl 2620903 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP31806" level="ERROR" subcomp="upgrade-coordinator"] Install of offline bundle failed on host ##.##.##.## with error : java.rmi.RemoteException:  [LiveInstallationError] VMware_bootbank_nsx-esx-datapath_#.#.#.#.#-#.#.########: Error in running [/etc/init.d/vdr-pre-upgrade stop upgrade]: Return code: 1 Output: stop upgrade begin Traceback (most recent call last):   File "/etc/init.d/vdr-pre-upgrade", line 90, in <module>     main()   File "/etc/init.d/vdr-pre-upgrade", line 86, in main     raise Exception("VDR Failed to perform full sync, %d" % status) Exception: VDR Failed to perform full sync, 0  It is not safe to continue. Please reboot the host immediately to discard the unfinished update.      cause = ('nsx-lcp-bundle(#.#.#.#.#-#.#.########)', 'vdr-pre-upgrade', 'Error in running [/etc/init.d/vdr-pre-upgrade stop upgrade]:\nReturn code: 1\nOutput: stop upgrade begin\nTraceback (most recent call last):\n  File "/etc/init.d/vdr-pre-upgrade", line 90, in <module>\n    main()\n  File "/etc/init.d/vdr-pre-upgrade", line 86, in main\n    raise Exception("VDR Failed to perform full sync, %d" % status)\nException: VDR Failed to perform full sync, 0\n')       vibs = ['VMware_bootbank_nsx-esx-datapath_#.#.#.#.#-#.#.########'] Please refer to the log file for more details.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX 4.x

VMware NSX-T Data Center 3.x

Cause

In some situations, the VDR kernel module and Local control plane sync may be in unhealthy state, sometimes after an incomplete upgrade of the NSX VIBs on the ESXi host. If the VDR sync is not in expected state, then the upgrade of the NSX VIBs will fail.

Resolution

This is a condition that may occur in a VMware NSX environment.

Workaround
Reboot the affected ESXi host and retry the NSX upgrade.