NSX edge upgrade fails with error "CommandFailedError: Command [\'mount\', \'/os_bak\'] returned non-zero"

Products

• VMware NSX

Issue/Introduction

• NSX Edge upgrade fails during the download OS phase with the following error:
  
  

Command ['mount', '/os_bak'] returned non-zero code 32: mount: /os_bak: special device /dev/disk/by-uuid/################ does not exist.

Environment

VMware NSX 9.1.1

Cause

A systemd-fsck race condition in _apply_os_filesystem() causes edge upgrade failures with exit code 8 ("device is mounted"). When maintenance mode mounts /os_bak, systemd spawns a background fsck service that continues running after unmount_device() completes. When this fsck finishes during the dd operation window, systemd automatically re-mounts the device just before the upgrade script's e2fsck call, causing failure.

Resolution

It is a known issue in VMware NSX, caused by race condition. Please retry edge upgrade from NSX UI and the retry should resolve it.