Certificate-Manager Fails to Make vCenter VMCA an Intermediate/Subordinate CA

Products

• VMware vCenter Server

Issue/Introduction

• Created a private key and CSR to request an Intermediate/Subordinate CA signing certificate.
• Used Certificate Manager to replace the existing VMCA Root signing CA certificate.
• VMCA certificate replacement succeeds, but fails to restart services and rolls back.
• Unable to issue a new Machine SSL certificate from the replaced VMCA Root signing CA certificate.
• Reverting to default self-signed VMCA certificates works

Environment

vCenter 8.x

Cause

Issues with the private key or CSR or CA template used to issue the requested Intermediate/Subordinate VMCA certificate.

Resolution

• Use the Certificate Manager Utility to have VMCA generate a certificate signing request (CSR) and specifying the path (/root) to write the CSR and key. Or refer to your own PKI tools’ documentation for the process of creating and signing the intermediate CA certificate and key pair.
  
  Note, To prevent dependency issues with vSphere startup the intermediate CA key should not be protected with a password for the import operation. 
  
  
• Create a Template for VMCA as a Subordinate CA. See This template is used when configuring VMCA to act as a Subordinate CA to your Microsoft CA.
  
  Note, vSphere's FIPS certificate only validates RSA key sizes of 2048 bits and 3072 bits.
  
  
• Build a certificate chain.pem file with the newly issued Intermediate/Subordinate VMCA certificate and CA chain.
  
  Format as follows:
  
  -----BEGIN CERTIFICATE-----
  <Certificate of VMCA>
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  <Certificate of intermediary CA>
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  <Certificate of Root CA>
  -----END CERTIFICATE-----
  
  
• Validate the contents of the chain.pem file:
  
  vcsa#  while openssl x509 -noout -text -fingerprint | grep -i 'subject:\|issuer:\|not\|fingerprint'; do :; done < /root/chain.pem
  
  example output:
  
  Issuer: C = US, ST = California, L = Palo Alto, O = PKI, OU = CA Certs, CN = INTERMEDIATECA
  Not Before: Month Day Time Year GMT
  Not After : Month Day Time Year GMT GMT
  Subject:  C = US, ST = California, L = Palo Alto, O = PKI, OU = CA Certs, CN =  SUBVMCA
  SHA1 Fingerprint=##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
  
  Issuer: C = US, ST = California, L = Palo Alto, O = PKI, OU = CA Certs, CN = ROOTCA
  Not Before: Month Day Time Year GMT
  Not After : Month Day Time Year GMT GMT
  Subject:  C = US, ST = California, L = Palo Alto, O = PKI, OU = CA Certs, CN =  INTERMEDIATECA
  SHA1 Fingerprint=##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
  
  Issuer: C = US, ST = California, L = Palo Alto, O = PKI, OU = CA Certs, CN = ROOTCA
  Not Before: Month Day Time Year GMT
  Not After : Month Day Time Year GMT GMT
  Subject:  C = US, ST = California, L = Palo Alto, O = PKI, OU = CA Certs, CN =  ROOTCA
  SHA1 Fingerprint=##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
• Use the Certificate Manager Utility to “Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate”.
  
  “Please provide valid custom certificate for Root.” – (Specify the certificate chain you created, /root/chain.pem)
  “Please provide valid custom key for Root.” – (Specify the key file, /root/vsphere-key-nopw.pem)
  
  
• In a separate shell windows monitor service restarts and certificate-manage logs for failures:
  
  Shell 2- vcsa# watch service-control --status
  
  Shell 3 - vcsa# tail -f /var/log/vmware/vmcad/certificate-manager.log
  
  
• Validate the VMCA certificate is replaced when certificate-manager reaches Status : 100% Completed [All tasks completed successfully]:
  
  vcsa#  /usr/lib/vmware-vmca/bin/certool --getrootca --server localhost | grep -A 1 -i 'serial\|subject\|issuer'
  
  example output:
  
   Serial Number:
              ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
  --
          Issuer: C=US, ST=California, L=Palo Alto, O=PKI, OU=CA Certs, CN=INTERMEDIATECA
          Validity
  --
          Subject: C=US, ST=California, L=Palo Alto, O=PKI, OU=CA Certs, CN=SUBVMCA
• Validate the vCenter Machine SSL certificate is updated:
  
  vcsa# # echo ^c | openssl s_client -connect localhost:443 | grep -i 'subject\|issuer'
  
  example output:
  
  subject=CN = vcenter.example, C = US, ST = California, L = Palo Alto, O = PKI, OU = CA Certs
  issuer=C = US, ST = California, L = Palo Alto, O = PKI, OU = CA Certs, CN = SUBVMCA
  
  
  
  

Additional Information

Intermediate CA mode, also referred to as Subordinate CA mode, offers the advantage of automating the deployment of trusted certificates to vSphere infrastructure components. Nonetheless, some organizations express concerns about issuing signing certificates in this manner, primarily due to potential impersonation of the organization and the possible lifespan of CA certificates. Mitigation strategies include the use of intermediate CA roots, certificate revocation lists, and even the creation of separate CA roots which are not generally trusted by organizational systems.

If you use VMCA as an intermediate CA, or use custom certificates, you might encounter significant complexity and the potential for a negative impact to your security, and an unnecessary increase in your operational risk.