VMware Avi Load Balancer Controller showing as not reachable in NSX-T UI

Products

VMware NSX VMware Avi Load Balancer

Issue/Introduction

The issue is seen with environment running VMware NSX and VMware Avi Load Balancer (ALB).
Post upgrade of NSX-T or certificate change on the ALB, the ALB is no longer accessible in the NSX-T UI.
Checking in var/log/proton/nsxapi.log you are presented with the below Error code is 94506 and Unable to find certificate chain:

<TIMESTAMP> INFO http-nio-127.0.0.1-7440-exec-29 PolicyALBControllerDeploymentFacadeImpl 5540 POLICY [nsx@6876 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="admin"] [ALB Controller] Error code is 94506 with error msg NSX Advanced Load Balancer Controller is not reachable. {0}

<TIMESTAMP> INFO http-nio-127.0.0.1-7440-exec-29 AlbControllerClusterServiceImpl 5540 POLICY [nsx@6876 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="admin"] Resource Access Exception while occurred while getting Cluster runtime. Error is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<manager-ip>/api/cluster/runtime/": PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

The error message observed within the NSX UI is "Avi Load Balancer Controller is not reachable (0)"

Environment

VMware NSX
VMware Avi Load Balancer

Cause

This is caused when certificates are changed on the controller with certificates using a not well known CA.

Resolution

This is expected behavior with NSX.

Workaround:
The following needs to be done whenever certificates are changed on the controller with a not well known CA:

1. Export the root CA certificate and the intermediate certificate from the ALB and upload them onto the NSX manager with SCP.

a. From ALB UI: Templates > Security > SSL/TLS Certificates
b. Root/Intermediate CA > export > export to clipboard
c. Save to a .pem file in a text editor

2. Run the below commands as root from the NSX-T Manager CLI where <ca-file-path> is to location of the certificate uploaded in step 1:

a. keytool -importcert -alias startssl -keystore /usr/lib/jvm/jre/lib/security/cacerts -storepass changeit -file <ca-file-path> b.Trust the certificate by typing yes:Trust this certificate? [no]: yes

If the above path is not found, please use the command below:
keytool -importcert -alias startssl -keystore /usr/java/jre/lib/security/cacerts -storepass changeit -file <ca-file-path>
c. sudo cp <ca-file-path> /usr/local/share/ca-certificates/
d. sudo update-ca-certificates
e. service proton restart

3. Log into the NSX UI: System > Appliances, and confirm that ALB is showing a stable cluster.

Note:
The certificate has to be imported on all three Managers.

Please note the workaround above to import the AVI CA certificate into NSX Manager using the keytool command is not supported in VCF 9.1. If you are running VCF 9.1, please open a case with Broadcom Technical Support and reference this KB article for assistance.

Additional Information

When attempting to import a certificate into the cacerts keystore, if you encounter the following error:

keytool error: java.lang.Exception: Certificate not imported, alias <startssl> already exists

This error occurs because a certificate with the alias <startssl> already exists in the cacerts keystore. Since each alias must be unique within the keystore,
you cannot import a new certificate using an existing alias.

Verification:
To check whether the alias already exists in the cacerts keystore, use the following keytool commands:

-List all certificates and search for the alias:

keytool -list -v -keystore /usr/java/jre/lib/security/cacerts -storepass changeit | grep -i alias

-Check details of a specific alias:

keytool -list -v -keystore /usr/java/jre/lib/security/cacerts -alias startssl

Resolution:
To resolve this error, you should use a different alias when importing the certificate. Follow these steps:

1.Backup the existing cacerts keystore:

cd /usr/java/jre/lib/security/
cp -p cacerts cacerts_backup

2.Import the certificate with a new alias:

keytool -importcert -alias startssl1 -keystore /usr/java/jre/lib/security/cacerts -storepass changeit -file <ca-file-path>

Replace <ca-file-path> with the path to your certificate file.

Note:
The alias is a unique identifier for each certificate within the keystore. Using a different alias ensures that there are no conflicts and allows the new certificate
to be imported successfully.

If you are contacting Broadcom support about this issue, please provide the following:

Ensure log date range covers the full date of the event(s) being investigated. When in doubt, retrieve logs for all time.
NSX Manager log bundles
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation

Handling Log Bundles for offline review with Broadcom support