vIDM Configurator Page 8443 Fails to Load Due to Corrupted idm-cacerts

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Loading the Appliance Configurator on port 8443 (https://<vIDM_FQDN>:8443) fails with the error "This site can't be reached".
  
  
  
  
• Executing ss -tulpn | grep 8443 indicates that port 8443 is not listening.
  
  
  
  
• Triggering global environment (vIDM) inventory sync from Aria Suite Lifecycle, fails with the error LCMVIDMIMPORT0011.
• The /opt/vmware/horizon/workspace/logs/workspace.log captures the following initialization and keystore errors:
  
  SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["https-jsse-nio2-8443"]]
        org.apache.catalina.LifecycleException: Protocol handler initialization failed

        Caused by: java.security.KeyStoreException: problem accessing trust store
                at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(Unknown Source)
                at java.base/javax.net.ssl.TrustManagerFactory.init(Unknown Source)
                at com.vmware.horizon.tomcat.CustomTrustManager.<init>(CustomTrustManager.java:41)
                ... 27 more
        Caused by: java.io.IOException: DER length more than 4 bytes: 109
                at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source)
                at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source)
                at org.bouncycastle.asn1.ASN1InputStream.readObject(Unknown Source)
                at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(Unknown Source)
                at java.base/java.security.KeyStore.load(Unknown Source)
                at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(Unknown Source)
                at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(Unknown Source)
                at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(Unknown Source)
                ... 30 more

Environment

VMware Identity Manager 3.3.7

Cause

The idm-cacerts file located in /usr/local/horizon/conf is corrupted or empty on the affected vIDM node

Resolution

1. Take non-memory snapshots of all vIDM nodes in the cluster.

2. If operating in a clustered environment and the idm-cacerts file is intact on peer nodes, copy the healthy file from an operational node to replace the corrupted file on the affected node.

3. If the idm-cacerts file is corrupted or empty on all nodes, or if the appliance operates as a standalone instance, manually recreate the file using the following sequence.

FIPS_JAR=$(ls -1r /usr/local/horizon/jre-endorsed/bc-fips-*.jar | head -1)
echo "FIPS_JAR=${FIPS_JAR}"                     # must not be empty
ls -la ${FIPS_JAR}                              # must exist

CA_PWD=$(cat /usr/local/horizon/conf/cakeystore.pass)
echo "Password length: ${#CA_PWD}"             # must be 32

ls -la /usr/java/jre-vmware/lib/security/cacerts   # source JKS must exist

FIPS_JAR=$(ls -1r /usr/local/horizon/jre-endorsed/bc-fips-*.jar | head -1)
CA_PWD=$(cat /usr/local/horizon/conf/cakeystore.pass)

# Backup

cp /usr/local/horizon/conf/idm-cacerts \
   /usr/local/horizon/conf/idm-cacerts.bad.$(date +%Y%m%d_%H%M%S)

# Delete corrupted file

rm -f /usr/local/horizon/conf/idm-cacerts

#Generate the new keystore:

/usr/java/jre-vmware/bin/keytool -importkeystore \
  -srckeystore  /usr/java/jre-vmware/lib/security/cacerts \
  -destkeystore /usr/local/horizon/conf/idm-cacerts \
  -srcstoretype JKS \
  -deststoretype BCFKS \
  -srcstorepass changeit \
  -deststorepass "${CA_PWD}" \
  -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \
  -providerPath ${FIPS_JAR}
 
#Verify the newly created keystore:

FIPS_JAR=$(ls -1r /usr/local/horizon/jre-endorsed/bc-fips-*.jar | head -1)
CA_PWD=$(cat /usr/local/horizon/conf/cakeystore.pass)

usr/java/jre-vmware/bin/keytool -list \
  -keystore /usr/local/horizon/conf/idm-cacerts \
  -storetype BCFKS \
  -storepass "${CA_PWD}" \
  -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \
  -providerPath ${FIPS_JAR} | head -4

4. Ensure the idm-cacerts file has the correct permissions applied:

chmod 644  /usr/local/horizon/conf/idm-cacerts
chown horizon:root  /usr/local/horizon/conf/idm-cacerts



5. Restart the Workspace service to apply the configuration:

service horizon-workspace restart