Replacing SDDC certificates with VMCA

Products

• VMware Cloud Foundation

Issue/Introduction

• SDDC manager UI is down
• Using CA certificate but certificate is expired when checked using below command

  /opt/vmware/sddc-support/sos --certificate-health

• If unable to generate certificate from CA server, momentarily to restore production we can replace SDDC certificates to VMCA signed.

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Resolution

1. Take a snapshot of SDDC manager VM and vCenter VM without memory (powered off snapshots of all vCenter VMs if in ELM)
2. SSH to SDDC Manager with vcf and su to root
3. Generate CSR on SDDC Manager using below command

   openssl req -new -newkey rsa:2048 -nodes -keyout sddc.key -out sddc.csr

    
4. SSH to the management vCenter and create a directory for these certificates

   mkdir /tmp/certs

5. Move the sddc.csr to the above path from SDDC manager to vCenter either using WINSCP or  VI
6. Create the certificate config file cert.cfg containing the SDDC Manager FQDN

   [ req ]
   req_extensions = v3_req
    
   [ v3_req ]
   extendedKeyUsage = serverAuth, clientAuth
   authorityKeyIdentifier=keyid,issuer
   authorityInfoAccess = caIssuers;URI:https://sddc-manager.example.com/afd/vecs/ca

7. Generate certificate using the cfg file and CSR

   openssl x509 -req -days 3650 -in sddc-manager.example.com.csr -out sddc-manager.example.com.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile cert.cfg

8. Create chain containing vCenter VMCA root and the certificate generated using below command

   cat sddc-manager.example.com.crt>>sddc-manager.example.com.pem
   cat /var/lib/vmware/vmca/root.cer>>sddc-manager.example.com.pem

9. On SDDC manager SSH session take backup of the older key and certificate

   cp /etc/ssl/private/vcf_https.key /etc/ssl/private/old_vcf_https.key
   cp /etc/ssl/certs/vcf_https.crt /etc/ssl/certs/old_vcf_https.crt

10. Upload sddc-manager.example.com.pem chain to /home/vcf
11. Replace the key and certificate with new one that was generated

    rm /etc/ssl/certs/vcf_https.crt
    mv /home/vcf/sddc-manager.example.com.pem /etc/ssl/certs/vcf_https.crt
    mv /home/vcf/sddc.key /etc/ssl/private/vcf_https.key

12. Assign permissions to crt and key file

    chmod 644 /etc/ssl/certs/vcf_https.crt
    chmod 640 /etc/ssl/private/vcf_https.key

13. Restart nginx service 

    nginx -t && systemctl reload nginx

Additional Information

• Check if VMCA root is in SDDC trust store, if not publish it using How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores
• If SDDC has expired store certificate delete it using https://docs.vmware.com/en/VMware-Cloud-Foundation/4.0/vcf-admin/GUID-8AC84B4C-5A58-481B-B70E-8616179B2C3F.html
• Delete expired trusted root certificates in vCenter if any using the KB Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)