Smarts NCM: NCM API/DS API Connection shows "Test Failed" in the Smarts NCM Web console; vc_smarts_adaptor.log contains "ERROR Public API Connection is DOWN" message

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:

The following conditions, events or system behaviors may be observed with this issue:

• Smarts NCM vc_smarts_adaptor.log contains "ERROR Public API Connection is DOWN" as shown in the following:

[2014-01-21 16:31:10,654] grails ERROR Public API Connection is DOWN. (com.voyence.integration.smarts.util.VoyenceConnectionBroker)

javax.naming.NamingException: Failed to retrieve Naming interface for provider https://NCM_hostname.local:8880/invoker/JNDIFactory [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

• NCM API/DS API Connection entry shows "Test Failed" in the Smarts NCM Web console interface as shown in the following:

Environment

VMware Smart Assurance - NCM

Cause

The above issue may be occurring because the certificate from the Primary Root CA (Certificate Authority) is missing or invalid in the java_cacerts file for your Smarts NCM environment.

Resolution

To investigate and address this issue, do the following:

1. View the voyence.conf configuration file:

cat /etc/voyence.conf

2. Note which JDK is in use for the environment.
3. Open the vc_smarts_adaptor.log log file in the /opt/smarts-ncm/logs directory.
4. Copy the HTTPS URL from the "ERROR Public API Connection is DOWN" error message (the HTTPS URL that Java could not access), and paste it into your browser. For example, the URL in the above log message example (copied again below) is:

https://NCM_hostname.local:8880/invoker/JNDIFactory

Complete example log message

[2014-01-21 16:31:10,654] grails ERROR Public API Connection is DOWN. (com.voyence.integration.smarts.util.VoyenceConnectionBroker)

javax.naming.NamingException: Failed to retrieve Naming interface for provider https://NCM_hostname.local:8880/invoker/JNDIFactory [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

5. Go to the URL and click the HTTPS certificate chain. In Internet Explorer, this chain will be represented by a lock icon. In Mozilla Firefox, it will be the domain name left of the URL.
6. Navigate the certificate hierarchy to the top. At the top of the hierarchy there should be a Primary Root CA (Certificate Authority). Note the certificate Issuer and Serial Number for the Primary Root CA.
7. From the command line, change (cd) to the /bin directory for the JDK that is currently in use in the environment as in the following example:

cd /usr/java/jdk1.7.0_45/bin

8. Run the keytool utility to see what certificates are in the keystore and output to a text file:

./keytool -list -v -keystore /usr/java/jdk1.7.0_45/jre/lib/security/cacerts >java_cacerts.txt

 

The output will redirect to the java_cacerts.txt file in the /bin directory where you ran the command. 

9. Open the java_cacerts.txt file and search for the serial number of the same certificate that is present in the browser for the Primary Root CA. In the java_cacerts.txt file, the serial number will be in lowercase and without the colon (:) character. If the matching serial number is not present, the Primary Root CA certificate is missing and will need to be imported as described in the following section.

Importing the Primary Root CA certificate to the Smarts NCM java_cacerts file
If the certificate from the Primary Root CA is not present in the Smarts NCM java_cacerts file, you can import the Primary Root CA certificate found in the browser to the Smarts NCM java_cacerts file as follows:

1. In the browser HTTPS certificate chain, export the Primary Root CA. Choose the "X.509 Certificate (DER)" type, so that the exported file has a *.der extension.
2. Choose the alias for the certificate file. Assuming the file is called example.der, choose the alias 'example' for the certificate.
3. Import the file using the following keytool utility command (assuming the preceding example.der file name and example alias):

keytool -import -alias example -keystore  /usr/java/jdk1.7.0_45/jre/lib/security/cacerts -file example.der

4. When prompted for a password, use 'changeit'.
5. Respond "yes" to trust this key and complete the certificate import.