VMware NSX-T Advanced Load Balancer Controller showing as not reachable in NSX-T UI

Products

VMware NSX

Issue/Introduction

Symptoms:

You are using VMware NSX-T and NSX Advanced Load Balancer (ALB).
After upgrade of NSX-T or certificate change on the ALB, the ALB is no longer accessible in the NSX-T UI.
Checking in var/log/proton/nsxapi.log you are presented with the below Error code is 94506 and Unable to find certificate chain:

2022-08-11T12:54:34.337Z INFO http-nio-127.0.0.1-7440-exec-29 PolicyALBControllerDeploymentFacadeImpl 5540 POLICY [nsx@6876 comp="nsx-manager" level="INFO" reqId="########-####-####-####-########49e8" subcomp="manager" username="admin"] [ALB Controller] Error code is 94506 with error msg NSX Advanced Load Balancer Controller is not reachable. {0}

2022-08-11T12:54:34.337Z INFO http-nio-127.0.0.1-7440-exec-29 AlbControllerClusterServiceImpl 5540 POLICY [nsx@6876 comp="nsx-manager" level="INFO" reqId="########-####-####-####-########49e8" subcomp="manager" username="admin"] Resource Access Exception while occurred while getting Cluster runtime. Error is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<manager-ip>/api/cluster/runtime/": PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

Cause

This is caused when certificates are changed on the controller with certificates using a not well known CA.

Resolution

This is not a bug and NSX-T is working as expected.

Workaround:
The following needs to be done whenever certificates are changed on the controller with a non well known CA:

1. Export the root CA certificate from the ALB and save onto the NSX manager.

2. Run the below commands as root from the NSX-T Manager CLI where <ca-file-path> is to location of the certificate uploaded in step 1:

a. keytool -importcert -alias startssl -keystore /usr/lib/jvm/jre/lib/security/cacerts -storepass changeit -file <ca-file-path>
If the above path is not found, please use the command below:
b. keytool -importcert -alias startssl -keystore /usr/java/jre/lib/security/cacerts -storepass changeit -file <ca-file-path>
c. sudo cp <ca-file-path> /usr/local/share/ca-certificates/
d. sudo update-ca-certificates
e. service proton restart

3. Log into the NSX-T UI and confirm the ALB page is now loading as expected

Additional Information

When attempting to import a certificate into the cacerts keystore, if you encounter the following error:

keytool error: java.lang.Exception: Certificate not imported, alias <startssl> already exists

This error occurs because a certificate with the alias <startssl> already exists in the cacerts keystore. Since each alias must be unique within the keystore,
you cannot import a new certificate using an existing alias.

Verification
To check whether the alias already exists in the cacerts keystore, use the following keytool commands:

-List all certificates and search for the alias:

keytool -list -v -keystore /usr/java/jre/lib/security/cacerts -storepass changeit | grep -i alias

-Check details of a specific alias:

keytool -list -v -keystore /usr/java/jre/lib/security/cacerts -alias startssl

Resolution
To resolve this error, you should use a different alias when importing the certificate. Follow these steps:

1.Backup the existing cacerts keystore:

cd /usr/java/jre/lib/security/
cp -p cacerts cacerts_backup

2.Import the certificate with a new alias:

keytool -importcert -alias startssl1 -keystore /usr/java/jre/lib/security/cacerts -storepass changeit -file <ca-file-path>

Replace <ca-file-path> with the path to your certificate file.

Note
The alias is a unique identifier for each certificate within the keystore. Using a different alias ensures that there are no conflicts and allows the new certificate
to be imported successfully.