NSX Upgrade Precheck fails with the error "Failed to create local backup on '[<nsx-manager-node-ip>]' node"

Products

• VMware NSX

Issue/Introduction

• Upgrade Prechecks fail with error message: "Failed to create local backup on '[<nsx-manager-node-ip>]' node"
• On the NSX Manager node marked by IP, the file /var/log/upgrade-coordinator/upgrade-coordinator.log includes lines similar to:

2025-02-26T20:24:09.504Z  INFO task-executor-9-1-workitem-MP-DataMigrationDryRun UpgradeServiceImpl 4512 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Component type = MP , Upgrade Status = IN_PROGRESS, Progress Percentage = 0 , Progress Message = Performing data migration dry-run...

Creating configuration backup

Creating configuration backup on <nsx-manager-node-ip> node

Successfully created configuration backup on <nsx-manager-node-ip> node

Creating local backup

Creating local backup on <nsx-manager-node-ip> node

2025-02-26T20:24:09.504Z  INFO task-executor-9-1-workitem-MP-DataMigrationDryRun UpgradeServiceImpl 4512 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Component type MP is in progress. Returning it as component in progress.

2025-02-26T20:24:09.504Z  INFO task-executor-9-1-workitem-MP-DataMigrationDryRun ExecutionMonitorServiceImpl 4512 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Reporting failure of upgrade unit DataMigrationDryRun in upgrade plan

2025-02-26T20:24:09.505Z  INFO pool-44-thread-1 UpgradeServiceImpl 4512 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Pause is in progress. Currently executor is executing 1 components, 0 groups and 0 single type tasks.

2025-02-26T20:24:09.505Z  INFO task-executor-8-1 ExecutionMonitorServiceImpl 4512 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Execution monitor service invoked to react to completion of upgrade of group PreUpgradeChecks

2025-02-26T20:24:09.505Z  INFO task-executor-8-1 ExecutionMonitorServiceImpl 4512 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Skip calling onGroupUpgradeComplete on plugin because upgrade status of group wasn't SUCCESS

2025-02-26T20:24:09.521Z  INFO task-executor-9-1-workitem-MP-DataMigrationDryRun ReferencedProgressCollectorImpl 4512 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Setting upgrade status for MP to FAILED, Total 7, Success 0, Fail 1, Paused 0, Not-started 6

2025-02-26T20:24:09.532Z ERROR task-executor-9-1-workitem-MP-DataMigrationDryRun MPRollingUpgradeServiceImpl 4512 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30459" level="ERROR" subcomp="upgrade-coordinator"] Failed to create local backup on '[<nsx-manager-node-ip>]' node

 

• In the same log, there is a stack trace with similar messages:

2025-02-26T20:24:09.494Z  WARN task-executor-9-1-workitem-MP-DataMigrationDryRun MPClusterUpgradeServiceImpl 4512 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] Local backup generation failed. Error - I/O error on POST request for "https://<nsx-manager-node-ip>:443/api/v1/upgrade/actions/take_local_backup": Certificate expired for CN=<objectfqdn>,OU=<CERT_OU>,O=<CERT_ORG>,L=<CERT_LOCATION>,ST=<CERT_STATE>,C=<CERT_COUNTRY>; nested exception is javax.net.ssl.SSLHandshakeException: Certificate expired for CN=<objectfqdn>,OU=<CERT_OU>,O=<CERT_ORG>,L=<CERT_LOCATION>,ST=<CERT_STATE>,C=<CERT_COUNTRY>.org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://<nsx-manager-node-ip>:443/api/v1/upgrade/actions/take_local_backup": Certificate expired for CN=<objectfqdn>,OU=<CERT_OU>,O=<CERT_ORG>,L=<CERT_LOCATION>,ST=<CERT_STATE>,C=<CERT_COUNTRY>; nested exception is javax.net.ssl.SSLHandshakeException: Certificate expired for CN=<objectfqdn>,OU=<CERT_OU>,O=<CERT_ORG>,L=<CERT_LOCATION>,ST=<CERT_STATE>,C=<CERT_COUNTRY>

 

Environment

VMware NSX-T Data Center

VMware NSX

Cause

This error occurs if the certificate in use by the NSX Manager node at port 443 is expired.

Resolution

Use the script attached to KB 369034 to automatically identify and replace all certificates in use on the NSX Manager with self-signed certificates.  After this is complete, attempt the upgrade prechecks again.