NSX Upgrade Failed at 'Install

Products

VMware NSX

Issue/Introduction

An NSX upgrade has been initiated. The services on the first NSX Manager node to be upgraded went down and the upgrade did not progress.
NSX CLI: Confirms that the cluster and nodes services remain down:
get cluster status
NSX UI System > Appliances (First NSX Manager Node to be upgraded):
- Service Status, CPU, Memory, Storage and Alarms may not display a reading
- Version is the current (pre-upgraded) NSX version
- Cluster status is 'degraded'.

NSX upgrade orchestrator node CLI confirms that the NSX upgrade failed at the 'install_os' step:
get upgrade progress-status

NB: To confirm upgrade orchestrator node, run the following command from any NSX Manager node:
get service install-upgrade

download_os [2026-02-25 11:10:25 - 2026-02-25 11:14:19] SUCCESS
>shutdown_manager [2026-02-25 11:14:22 - 2026-02-25 11:15:34] SUCCESS
>install_os [2026-02-25 11:15:34 - 2026-02-25 11:17:22] FAILED
>install_os [2026-02-25 12:46:36 - 2026-02-25 12:51:35] FAILED
>------ Output of last step start ------
>Stderr: An unexpected exception occurred: CommandFailedError: Command ['strace', '-o', '/var/log/vmware_mount_trace.log', '/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/common/vmware-mount/vmware-mount', '-r', '/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/files/target.vmdk', '2', '/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/files/mount-point'] returned non-zero code 1: b''
>Troubleshooting: Upgrade has failed. Appliance OS is of old version, however, UI will not be available. Please contact GSS to retry the upgrade.
------ Output of last step end ------

NSX upgrade orchestrator node logs confirm that unmount failed with error 'target is busy' after OS filesystem is copied:
/var/log/syslog

2026-02-25T11:15:42.936Z <NSX Manager (Orchestrator) Node> NSX 3558174 SYSTEM [nsx@4413 comp="nsx-manager nsx-policy-manager nsx-controller" subcomp="upgrade-bundle" level="INFO"] Successfully mounted /image/VMware-NSX-unified-appliance-9.0.2.0.25150388/files/target.vmdk to /image/VMware-NSX-unified-appliance-9.0.2.0.25150388/files/mount-point

2026-02-25T11:15:42.035Z <NSX Manager (Orchestrator) Node> NSX 3558174 SYSTEM [nsx@4413 comp="nsx-manager nsx-policy-manager nsx-controller" subcomp="upgrade-bundle" level="INFO"] Applying OS filesystem from /image/VMware-NSX-unified-appliance-9.0.2.0.25150388/files/mount-point to /os_bak

2026-02-25T11:16:52.729Z <NSX Manager (Orchestrator) Node> NSX 3558174 SYSTEM [nsx@4413 comp="nsx-manager nsx-policy-manager nsx-controller" subcomp="upgrade-bundle" level="INFO"] Copied OS contents

2026-02-25T11:16:52.091Z <NSX Manager (Orchestrator) Node> systemd 1 - -  image-VMware\x2dNSX\x2dunified\x2dappliance\x2d9.0.2.0.25150388-files-mount\x2dpoint.mount: Deactivated successfully.

2026-02-25T11:16:52.539Z <NSX Manager (Orchestrator) Node> NSX 3558174 SYSTEM [nsx@4413 comp="nsx-manager nsx-policy-manager nsx-controller" subcomp="upgrade-bundle" level="INFO"] Unmount attempt 0 of /image/VMware-NSX-unified-appliance-9.0.2.0.25150388/files/mount-point failed with error Command ['/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/common/vmware-mount/vmware-mount', '-x'] returned non-zero code 1: b'umount: /var/run/vmware/fuse/15860554906018132400.links/19: target is busy.\numount: /var/run/vmware/fuse/15860554906018132400.links/19: target is busy.\n'

2026-02-25T11:17:04.666Z <NSX Manager (Orchestrator) Node> NSX 3558174 SYSTEM [nsx@4413 comp="nsx-manager nsx-policy-manager nsx-controller" subcomp="upgrade-bundle" level="INFO"] Unmount attempt 1 of /image/VMware-NSX-unified-appliance-9.0.2.0.25150388/files/mount-point failed with error Command ['/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/common/vmware-mount/vmware-mount', '-x'] returned non-zero code 1: b'umount: /var/run/vmware/fuse/15860554906018132400.links/19: target is busy.\numount: /var/run/vmware/fuse/15860554906018132400.links/19: target is busy.\n'

2026-02-25T11:17:22.148Z <NSX Manager (Orchestrator) Node> NSX 3558174 SYSTEM [nsx@4413 comp="nsx-manager nsx-policy-manager nsx-controller" subcomp="upgrade-bundle" level="ERROR"] Command ['/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/common/vmware-mount/vmware-mount', '-x'] returned non-zero code 1: b'umount: /var/run/vmware/fuse/15860554906018132400.links/19: target is busy.\numount: /var/run/vmware/fuse/15860554906018132400.links/19: target is busy.\n'#012Traceback (most recent call last):#012  File "/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/src/task.py", line 202, in run#012    self.run_action()#012  File "/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/src/task.py", line 103, in wrapper#012    func(self, *args, **kwargs)#012  File "/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/src/task.py", line 754, in run_action#012    self.start_os_install()#012  File "/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/src/task.py", line 789, in start_os_install#012    self._apply_os_filesystem()#012  File "/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/src/task.py", line 986, in _apply_os_filesystem#012    self.unmount_vmdk(vmware_mount, mount_point, 3)#012  File "/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/src/task.py", line 1009, in unmount_vmdk#012    raise e#012  File "/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/src/task.py", line 997, in unmount_vmdk#012    host.run_cmd([vmware_mount, '-x'])#012  File "/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/src/host.py", line 206, in run_cmd#012    raise error.CommandFailedError(#012error.CommandFailedError: Command ['/image/VMware-NSX-unified-appliance-9.0.2.0.25150388/common/vmware-mount/vmware-mount', '-x'] returned non-zero code 1: b'umount: /var/run/vmware/fuse/15860554906018132400.links/19: target is busy.\numount: /var/run/vmware/fuse/15860554906018132400.links/19: target is busy.\n'

Environment

VMware NSX

Cause

The issue is result of a failure with the vmware-mount tool which is scheduled for replacement in a later version of NSX.

Resolution

This is a known issue impacting VMware NSX.

Workaround

Rollback upgrade for NSX Manager nodes.
Remove the failed NSX Manager node.
Add a new (replacement) NSX Manager node.
Proceed with the NSX upgrade.