NSX manager Node upgrade failed: MP upgrade failed at start_manager step with error:

Products

• VMware NSX

Issue/Introduction

NSX Manager upgrade is failing at start_manager step.

"Unexpected error while upgrading upgrade-unit: [MPP] Node upgrade failed : MP upgrade failed at start_manager step with error : /image/VMware-NSX-unified- appliance-4.2.3.1.0.24954571/scripts/start_manager.py:12: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives from distutils.version import LooseVersion ..

From NSX UI> Upgrade view:

From SDDC UI: MP upgrade failed due to timeout waiting for step reboot to complete.

 

> get upgrade progress-status <--from the admin cli of the failed manager node.

Upgrade info:
From-version: 4.2.2.1.0.24765087
To-version: 4.2.3.1.0.24954571

Upgrade steps:(Timelines)
download_os [2025-10-16 13:36:32 - 2025-10-16 13:36:57] SUCCESS
shutdown_manager [2025-10-16 13:47:10 - 2025-10-16 13:48:40] SUCCESS
install_os [2025-10-16 13:48:40 - 2025-10-16 13:49:36] SUCCESS
migrate_manager_config [2025-10-16 13:49:37 - 2025-10-16 13:49:42] SUCCESS
pin_ui [2025-10-16 13:49:42 - 2025-10-16 13:49:47] SUCCESS
switch_os [2025-10-16 13:49:47 - 2025-10-16 13:49:48] SUCCESS
reboot [2025-10-16 13:49:48 - 2025-10-20 07:17:15] SUCCESS
start_manager [2025-10-17 05:16:42 - 2025-10-17 05:18:18] FAILED <-----------*************

------ Output of last step start ------
    Status:
Error unmasking nsx-custom target: b'Failed to get properties: Transport endpoint is not connected\n', return value 1

Stdout: Removed /config/.resume_upgrade flag
Starting start_manager upgrade script
Error unmasking nsx-custom target: b'Failed to get properties: Transport endpoint is not connected\n', return value 1

Stderr: /image/VMware-NSX-unified-appliance-4.2.3.1.0.24954571/scripts/start_manager.py:12: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.version import LooseVersion

 

 

/var/log/upgrade-coordinator/upgrade-coordinator.log:

YYYY-MM-DD HH:MM:SS.ms:  INFO Thread-8 MPRollingUpgradeServiceImpl 75212 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Playbook resume response for node 84153142-####-####-####-4a12bf984469 is [UC] Error in rest call. url= /api/v1/cluster/84153142-####-####-####-4a12bf984469/node/upgrade/performtask?action=execute , method= POST , response= {"error_code": 36580, "error_message": "Error proxying request to: 84153142-####-####-####-4a12bf984469.", "module_name": "node-services"} , error= 500 Internal Server Error: "{"error_code": 36580, "error_message": "Error proxying request to: 84153142-####-####-####-4a12bf984469.", "module_name": "node-services"}" .

YYYY-MM-DD HH:MM:SS.ms  INFO task-executor-4-1-workitem-MP-84153142-####-####-####-4a12bf984469 MPRollingUpgradeServiceImpl 75212 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Upgrade progress status is UpgradeProgressStatus{upgradeBundlePresent=true, lastUpgradeStepStatus=LastUpgradeStepStatus{stepName='start_manager', startTime='YYYY-MM-DD HH:MM:SS.ms:', endTime='null', failureReason='null', state='TASK_IN_PROGRESS', stdErr='/image/VMware-NSX-unified-appliance-4.2.3.1.0.24954571/scripts/start_manager.py:12: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

Environment

VMware NSX

Cause

Due to temporary communication issue within the systemd control group, the systemd D-Bus interface was momentarily unavailable or unstable.

Resolution

Retry the failed upgrade task from the UI.