VMware NSX-T Advanced Load Balancer Controller showing as not reachable in NSX-T UI

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You are using VMware NSX-T and NSX Advanced Load Balancer (ALB).
• After upgrade of NSX-T or certificate change on the ALB, the ALB is no longer accessible in the NSX-T UI.
• Checking in var/log/proton/nsxapi.log you are presented with the below Error code is 94506 and Unable to find certificate chain:

2022-08-11T12:54:34.337Z INFO http-nio-127.0.0.1-7440-exec-29 PolicyALBControllerDeploymentFacadeImpl 5540 POLICY [nsx@6876 comp="nsx-manager" level="INFO" reqId="11cd8578-dbdf-4bcb-9bd3-35d7e02c49e8" subcomp="manager" username="admin"] [ALB Controller] Error code is 94506 with error msg NSX Advanced Load Balancer Controller is not reachable. {0}

2022-08-11T12:54:34.337Z INFO http-nio-127.0.0.1-7440-exec-29 AlbControllerClusterServiceImpl 5540 POLICY [nsx@6876 comp="nsx-manager" level="INFO" reqId="11cd8578-dbdf-4bcb-9bd3-35d7e02c49e8" subcomp="manager" username="admin"] Resource Access Exception while occurred while getting Cluster runtime. Error is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://172.16.124.50/api/cluster/runtime/": PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.
 

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

Cause

This is caused when certificates are changed on the controller with certificates using a not well known CA.

Resolution

This is not a bug and NSX-T is working as expected.

Workaround:
The following needs to be done whenever certificates are changed on the controller with a non well known CA:

1. Export the root CA certificate from the ALB and save onto the NSX manager.

2. Run the below commands as root from the NSX-T Manager CLI where <ca-file-path> is to location of the certificate uploaded in step 1:

a. keytool -importcert -alias startssl -keystore /usr/lib/jvm/jre/lib/security/cacerts -storepass changeit -file <ca-file-path>
If the above path is not found, please use the command below:
b. keytool -importcert -alias startssl -keystore /usr/java/jre/lib/security/cacerts -storepass changeit -file <ca-file-path>
c. sudo cp <ca-file-path> /usr/local/share/ca-certificates/
d. sudo update-ca-certificates
e. service proton restart

3. Log into the NSX-T UI and confirm the ALB page is now loading as expected