vSphere Update Manager in Aria Orchestrator Inventory is stuck loading / Any Endpoint registration fails with the error: 'Server certificate chain is not trusted, and the thumbprint doesn't match'.

Products

• VMware Aria Suite

Issue/Introduction

vSphere Update Manager fails to load in the Aria Orchestrator inventory.
Additionally, the 'Set a default vCenter with Update Manager' workflow is unable to retrieve a list of available vCenter instances.
However, the 'Add a vCenter Server instance' workflow successfully adds vCenter servers to the Aria Orchestrator inventory.

Example View:

The /services-logs/prelude/vco-app/file-logs/vco-server-app.log file contains errors similar to:

Caused by: com.vmware.o11n.vmo.plugin.vmware_update_manager.exception.VumClientException: Cannot create a VUM client.
        at com.vmware.o11n.vmo.plugin.vmware_update_manager.connectivity.ClientManager.getVumClient(ClientManager.java:199) ~[?:?]
        at com.vmware.o11n.vmo.plugin.vmware_update_manager.VumServiceConfigurator.getVumClient(VumServiceConfigurator.java:38) ~[?:?]
        at com.vmware.o11n.vmo.plugin.vmware_update_manager.DefaultServiceFactory.initService(DefaultServiceFactory.java:121) ~[?:?]
--
Caused by: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261) ~[vlsi-client-1.0.0.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:356) ~[vlsi-client-1.0.0.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:57) ~[vlsi-client-1.0.0.jar:?]
--
Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match
        at com.vmware.vim.vmomi.client.http.impl.ClientExceptionTranslator.translate(ClientExceptionTranslator.java:75) ~[vlsi-client-1.0.0.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261) ~[vlsi-client-1.0.0.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:356) ~[vlsi-client-1.0.0.jar:?]
--
Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: SSL handshake from 0.0.0.0/0.0.0.0:59216 to FQDN/x.x.x.x:443 failed in 4 ms
        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.handleHandshakeException(ThumbprintTrustManager.java:649) ~[vlsi-client-1.0.0.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.verify(ThumbprintTrustManager.java:474) ~[vlsi-client-1.0.0.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.verifyHostname(VlsiSslSocketFactory.java:129) ~[vlsi-client-1.0.0.jar:?]

Environment

Aria Orchestrator 8.18.x

Resolution

Add the Vcenter certificate to the VCO keystore and reconfigure VRO container.

Kindly make sure to have a snapshot / backup of the Aria Orchestrator appliances prior following the steps below:

To add the certificate to the vco keystore, please follow the below steps:

1. Get the keystore password

    kubectl get vaconfig -o json | jq ".items[0].spec.vro.keystorePass" -r

2. Go inside the VCO container 

    a) Find the pod name:

      kubectl -n prelude get pods | grep vco-app 

   b) Access the VCO container using the below command: 

     kubectl exec -it -n prelude <vco-app-xxx> -c vco-server-app -- /bin/bash

3. Obtain the Vcenter certificate:

echo -n | openssl s_client -connect <vc-fqdn>:443 -showcerts | tac | sed -ne '1,/-BEGIN CERTIFICATE-/p' | tac | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | tee -a vCentercertificate.crt

4. Import the vCenter certificate to VCO keystore:

keytool -keystore /usr/lib/vco/app-server/conf/security/jssecacerts -storepass <keystore-password-from-step1> -importcert -file helloo.crt -alias AAcert

Steps to reconfigure VRO container 

1. Get the name of vco-app pod:

# kubectl -n prelude get pods | grep vco-app
vco-app-8ffdxxxxx-xxxxx     

2. Access the VCO container using the below command:

kubectl exec -it -n prelude <vco-app-xxx> -c vco-server-app -- /bin/bash

3. Modify setenv.sh file using the nano editor:

nano /usr/lib/vco/app-server/bin/setenv.sh

4. Find #vRO system properties and add 2 lines:

-Djavax.net.ssl.trustStore=${VCO_INSTALL_PATH}/app-server/conf/security/jssecacerts \
-Djavax.net.ssl.trustStorePassword=${KS_PASSWORD} \

Example:
{code:java}
# vRO system properties
JVM_OPTS="$JVM_OPTS \
-Djava.awt.headless=true \
-Dch.dunes.install-path=${VCO_INSTALL_PATH} \
-Dlog4j.configurationFile=${VCO_INSTALL_PATH}/app-server/conf/log4j2.xml \
-Djavax.net.ssl.trustStore=${VCO_INSTALL_PATH}/app-server/conf/security/jssecacerts \
-Djavax.net.ssl.trustStorePassword=${KS_PASSWORD} \
-Dks.password=${KS_PASSWORD} \
-Dcom.sun.management.jmxremote \
--add-exports java.xml.crypto/org.jcp.xml.dsig.internal.dom=java.base \
"

5. Run kill command to kill the process and restart it:

kill 1 

 

6. Repeat the same steps on any other vco-app pods within the cluster. 

The above action plan is applicable for resolving an issue where the "Endpoint registration fails with the error: 'Server certificate chain is not trusted, and the thumbprint doesn't match'."

Additional Information

If above action plan does not work, please follow the steps below.

• Remove all attached vCenters from the vCenter plug-in (not the VUM plug-in). Refer to the screenshot for guidance.

• Verify removal of all vCenter and VUM endpoints from the following paths:

  • Resources > Library > VC

  • Resources > Library > VUM Refer to the screenshot.

• Manually delete all certificates associated with the removed vCenter endpoints.
  Use the following workflow:
  Workflows > Library > Configuration > SSL Trust Manager > Delete a trusted certificate
  Refer to the third screenshot to identify which certificate to delete. (Example: ######.#######.###)

• Re-attach each vCenter one by one. During each attachment, you will be prompted to accept the certificate.

• After attaching each vCenter, go to the Inventory and try to expand the VUM plug-in to verify if the issue persists.