CERT_VALIDATION_ERROR and Internal Server Error during vCenter Import - VCF Operations

Products

• VCF Operations
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• vCenter import task in VCF Operations Manager fails with error: InternalServerError
  
  
  
  
• Under any Browser's More Tools > Developer Tools > Network, below error stacks are found:
  
  A PUBLIC_INTERNAL_SERVER_ERROR occurs during the workload domain import precheck phase in VMware Cloud Foundation.
  

  {
    "errorCode": "PUBLIC_INTERNAL_SERVER_ERROR",
    "arguments": [],
    "message": "InternalServerError",
    "causes": [
        {
            "type": "org.springframework.web.client.HttpServerErrorException$InternalServerError",
            "message": "500 : \"{\"errorCode\":\"CERT_VALIDATION_ERROR\",\"arguments\":[],\"message\":\"Error while validating certificate\",\"causes\":[{\"type\":\"java.security.cert.CertificateExpiredException\",\"message\":\"NotAfter: <EXPIRY DATE>"}],\"referenceToken\":\"#####\"}\""
        }
    ],
    "referenceToken": "#####"
}

• Under the SDDC Manager's /var/log/vmware/vcf/domainmanager/domainmanager.log The following error stacks appear:
  
  ERROR [vcf_dm,05e01578c0b146c4,858b] [c.v.v.v.c.v.BrownfieldImportController,http-nio-127.0.0.1-7200-exec-10]  500 : "{"errorCode":"CERT_VALIDATION_ERROR","arguments":[],"message":"Error while validating certificate","causes":[{"type":"java.security.cert.CertificateExpiredException","message":"NotAfter: <EXPIRY DATE>"}],"referenceToken":"#####"}"

Environment

VCF 9.x

Cause

This issue occurs when an expired vCenter or internal certificate resides in the SDDC Manager trust store, causing the Spring Boot validation service to fail during the import precheck

Resolution

Manually remove the expired certificate from the SDDC Manager trust store.

1. Take a snapshot of the SDDC Manager Appliance without virtual machine memory.
2. SSH to the SDDC Manager Appliance as the vcf user and then issue a su - command to switch to the root user
3. Retrieve the password for the trusted certificate store:
   
   KEY=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)
   
   Run the following command and note down the password: 
   
   echo $KEY

    
4. List the entries in the keystore to locate the expired certificate alias:
   
   keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $KEY
   
    
5. Delete the expired certificate alias identified in the previous step:
   
   keytool -delete -alias #### -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store
   
    
6. Restart the services to flush the memory cache:
   
   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
   
   
7. Wait for services to stabilize, and then re-run the vCenter import workflow.