VMware NSX-T manager prechecks fail - Unable to retrieve disk space information on directory /tmp

Products

VMware NSX

Issue/Introduction

During an NSX-T upgrade the prechecks fail for the managers with error:

Failed to execute Check for sufficient free space on /tmp partition.
Failed to execute Check for sufficient free space on /image partition.
Failed to execute Check for sufficient free space on /config partition.

As root on the NSX-T managers, running df -h, we see no space issues.

/tmp has more than 18 MB free
/image has more than 4299 MB free
/config has more than 14796 MB free

In the NSX-T manager logs /var/log/upgrade-coordinator/upgrade-coordinator.log the following log ERROR is seen:

2025-##-0#T##:##:##.#### ERROR pool-48-thread-10 UcRestClient 240320 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30014" level="ERROR" subcomp="upgrade-coordinator"] Error during GET rest request /nsxapi/api/v1/cluster/nodes/#######-####-####-####-############/status?source=realtime , trial 2 , err com.vmware.nsx.management.upgrade.rpcframework.UcRestRpcException: [UC] Error in rest call. url= //nsxapi/api/v1/cluster/nodes/#######-####-####-####-############/status?source=realtime , method= GET , response= {
"module_name" : "common-services",
"error_message" : "General error has occurred.",
"details" : "Unable to reach client #######-####-####-####-############, application AggSvc",
"error_code" : 100
}

In the NSX-T manager log /var/log/upgrade-coordinator/upgrade-coordinator.log, this is the manager running the upgrade service (Orchestrator node),the following log WARNING's are seen:

less upgrade-coordinator.log | grep "failed nodes"
2025-##-0#T##:##:##.#### INFO pool-48-thread-10 MpFreeSpaceInspectionTask 240320 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] disk space /tmp failed nodes {CLUSTER_API_NOT_WORKING=[nsxt-mgr.######, nsxt-mgr.######]}
2025-##-0#T##:##:##.#### INFO pool-48-thread-10 MpFreeSpaceInspectionTask 240320 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] disk space /image failed nodes {CLUSTER_API_NOT_WORKING=[nsxt-mgr.######, nsxt-mgr.######]}
2025-##-0#T##:##:##.#### INFO pool-48-thread-10 MpFreeSpaceInspectionTask 240320 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] disk space /config failed nodes {CLUSTER_API_NOT_WORKING=[nsxt-mgr.######, nsxt-mgr.######]}

In the NSX-T manager logs var/log/vmware/appl-proxy-rpc.log the following log ERROR is seen:

grep "alert unsupported certificate (SSL routines" var/log/vmware/appl-proxy-rpc.log

2025-##-0#T##:##:##.#### nsx-mgr NSX 1955 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="2000" level="WARNING"] StreamConnection[9609 Connected to ssl://##.##.##.##:1234 sid:9610] ReadCallback - closing connection (error: 167773203-sslv3 alert unsupported certificate (SSL routines), socket: open)
2025-##-0#T##:##:##.#### nsx-mgr NSX 1955 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="2000" level="INFO"] StreamConnection[9609 Closed to ssl://##.##.##.##:1234 sid:-1] Closed (reason: network error, error: 167773203-sslv3 alert unsupported certificate (SSL routines))
2025-10-08T19:31:04.977Z nsx-mgr NSX 1955 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-rpc" tid="2000" level="WARNING"] RpcConnection[9609 Negotiating to ssl://nsxt-mgr.######:1234 0] ReadCallback - closing connection (error: 167773203-sslv3 alert unsupported certificate (SSL routines))
2025-##-0#T##:##:##.#### nsx-mgr NSX 1955 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="2000" level="WARNING"] StreamConnection[9611 Connected to ssl://##.##.##.##:1234 sid:9612] ReadCallback - closing connection (error: 167773203-sslv3 alert unsupported certificate (SSL routines), socket: open)
2025-##-0#T##:##:##.#### nsx-mgr NSX 1955 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="2000" level="INFO"] StreamConnection[9611 Closed to ssl://##.##.##.##:1234 sid:-1] Closed (reason: network error, error: 167773203-sslv3 alert unsupported certificate (SSL routines))

When the APH cert from a manager node is pulled, only Server Auth is seen in the Key usage:

openssl x509 -in etc/vmware/nsx-appl-proxy/appl-proxy-ar-cert.pem -noout -purpose
Certificate purposes:
SSL client : No
SSL client CA : No
SSL server : Yes
SSL server CA : No
Netscape SSL server : Yes
Netscape SSL server CA : No
S/MIME signing : No
S/MIME signing CA : No
S/MIME encryption : No
S/MIME encryption CA : No
CRL signing : No
CRL signing CA : No
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : No
Time Stamp signing : No
Time Stamp signing CA : No

Environment

VMware NSX

Cause

APH-TN certificate has a wrong Extended Key Usage value. It should have both Server and Client Authentication. The upgrade will fail if either one is missing as the connectivity gets broken between APH-APH (MP-MP).

This leads to the prechecks

GET API call failing.

/nsxapi/api/v1/cluster/nodes/########-09f2-53ba-61a4-############/status?source=realtime

This GET API return the status of the nodes for the prechecks, which include disk space and usage results.

Resolution

Custom APH-TN certificate

Custom APH-TN certificate should be replaced with a correct certificate which has both Server and Client Auth in Extended Key usage. Request a certificate with both Server Authentication and Client Authentication Extended Key Usages (EKUs).

Following steps can be used to import and replace it:

1. Login to manager UI. Go to System > Certificates > Import > Certificate

a. Name the certificate
b. Disable Service Certificate toggle
c. Include each node's UUID to the CN field to maintain the uniqueness
d. Click on save

2. Obtain the certificate id for the newly imported certificate from UI. The ID field should have the UUID.

3. Run the below API with the certificate id and node-id of one of the NSX-manager nodes:

POST https://<nsx-mgr>/api/v1/trust-management/certificates/<cert-id>?action=apply_certificate&service_type=APH_TN&node_id=<node-id>

Once certificate is replaced on all nodes, upgrade can be resumed.

OR

Self Signed APH-TN certificate

Self signed certificates can be generated through NSX GUI System -> Settings, Certificates
Obtain the certificate id for the newly created certificate from UI. The ID field should have the UUID.
Run the below API with the certificate id and node-id of one of the NSX-manager nodes:

POST https://<nsx-mgr>/api/v1/trust-management/certificates/<cert-id>?action=apply_certificate&service_type=APH_TN&node_id=<node-id>

Once certificate is replaced on all nodes, upgrade can be resumed.