Error: 'Certificate authorities update failed' when trying to Configure a Certificate Authority in vCF Operations

Products

• VCF Operations
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• An attempt to Configure a Certificate Authority for VMware Cloud Foundation fails with the following error:
  
  Certificate authorities update failed.


  
  
• On the SDDC Manager Operation Manager logs we are observing the following error indicating the destination Microsoft CA Server is missing and the TLS connection is not trusted:
  
  

  /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  yyyy-mm-ddThh:mm:ss.zzz+0000 ERROR [vcf_om,<task_id>] [c.v.v.c.ca.plugin.MicrosoftCaService,http-nio-127.0.0.1-7300-exec-5] 
  Unable to retrieve Microsoft server certificate as provided host <MSCA_FQDN> is not valid/reachable.

  
  
  
• On a different scenario the same error can be observed if the TCP port 443 is blocked between the VCF Operations Fleet Management Appliance and the destination Microsoft CA. This can be tracked in the Fleet Management logs:
  
  

  /var/log/vrlcm/vmware_vrlcm.log
  yyyy-mm-ddThh:mm:ss.zzz+0000 ERROR vrlcm[1253] [http-nio-8080-exec-10] [c.v.v.l.l.c.MSCARestClient]  -- Exception occurred while trying to validate Microsoft CA
  org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<MSCA_FQDN>/certsrv": Connect to <MSCA_FQDN>:443 [<MSCA_FQDN>/<MSCA_IP>] failed: 
  Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <MSCA_FQDN>:443 [<MSCA_FQDN>/<MSCA_IP>] failed: Connection timed out (Connection timed out)
  ..
  INFO vrlcm[1253] [http-nio-8080-exec-10] [c.v.v.l.c.l.MaskingPrintStream]  -- * SYSOUT/SYSERR CAPTURED:  -- Exception occurred while trying to validate Microsoft CA

Environment

VCF 9.x

Resolution

• If the Microsoft CA Server is missing and not trusted from the SDDC manager follow the steps below to add it manually to the key store

1. 1. Take a Snapshot of the SDDC Manager VM:
      Before proceeding, it is highly recommended to take an offline snapshot of the SDDC Manager VM.
      This provides a rollback point if any issues occur during the trust synchronization process.
      
      
   2. SSH into the SDDC Manager with vcf user, and elevate to root with command 'su -'
      
      
   3. Trust the certificate to the store by executing the following command:
      
      echo off  | openssl s_client -connect <msca_fqdn>:443  2>&1 | openssl x509 -outform PEM 1> /tmp/depot.crt  && keytool -importcert -alias <msca_alias> -file /tmp/depot.crt -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit
      
      If there is a proxy in place:
      
      echo off  | openssl s_client -proxy <proxy-fqdn>:<proxy port> -connect <msca_fqdn>:443  2>&1 | openssl x509 -outform PEM 1> /tmp/depot.crt  && keytool -importcert -alias <msca_alias> -file /tmp/depot.crt -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit
      
      
      Note: In both commands make sure to replace in the fields the correct values for the MSCA and the Proxy Server.
      
      
   4. Restart the SDDC Manager services:  
      
      echo 'y' | /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
      
      
   5. Re-attempt configuring Certificate Authority in vCF Instance under Certificates page in vCF Operations UI.
      
      

• If the error occurs because of TCP port 443 is blocked between the VCF Operations Fleet Management Appliance and the destination Microsoft CA engage your security team to allow the traffic.

Additional Information

Troubleshooting Microsoft CA configuration failures in VCF Operations 9.0.x
Unable to configure Microsoft Certificate Authority in SDDC Manager
Configure the Microsoft Certificate Authority for Basic Authentication
Configure a Certificate Authority for VMware Cloud Foundation
How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores