Replacing SDDC certificates with VMCA

Products

• VMware Cloud Foundation

Issue/Introduction

• SDDC manager UI is down or If you are unable to generate a csr from the SDDC UI
• Using CA certificate but certificate is expired when checked using below command

  /opt/vmware/sddc-support/sos --certificate-health

• If unable to generate certificate from CA server, momentarily to restore production we can replace SDDC certificates to VMCA signed.

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Resolution

Note: It is strongly recommended to take a snapshot of the SDDC Manager VM before proceeding with the steps below.

Script Method:

For the scripted method please download the script attached to the bottom of this article and run it as root user on the sddc manager appliance in the vcf/home directory:

1. ssh to sddc manager and su to root
2. transfer the script to home directory
3. run script ./renew_sddc_cert_with_vmca.sh

After running this script sometimes it might fail with the error : /bin/bash^M: bad interpreter: No such file or directory

Then execute the below command & rerun the script.

sed -i -e 's/\r$//' renew_sddc_cert_with_vmca.sh

 

Manual Method:

1. SSH to SDDC Manager with vcf and su to root
2. Generate the CSR on the SDDC Manager by executing the command below. It's recommended to run this command from the /home/vcf directory:
   

   openssl req -new -newkey rsa:2048 -nodes -keyout sddc-manager.example.com.key -out sddc-manager.example.com.csr

    
3. SSH to the management vCenter and create a directory for these certificates

   mkdir /tmp/certs

4. Move the sddc-manager.example.com.csr to the above path from SDDC manager to vCenter either using WINSCP or  VI
5. Create the certificate config file cert.cfg containing the MGMT vCenter FQDN

   [ req ]
   req_extensions = v3_req
    
   [ v3_req ]
   extendedKeyUsage = serverAuth, clientAuth
   authorityKeyIdentifier=keyid,issuer
   authorityInfoAccess = caIssuers;URI:https://mgmt-vcenter.example.com/afd/vecs/ca
   subjectAltName=IP:<SDDC ip_address>,DNS:<SDDC FQDN>

6. Generate certificate using the cfg file and CSR

   openssl x509 -req -days 3650 -in sddc-manager.example.com.csr -out sddc-manager.example.com.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile cert.cfg

7. Create chain containing vCenter VMCA root and the certificate generated using below command

   cat sddc-manager.example.com.crt>>sddc-manager.example.com.pem
   cat /var/lib/vmware/vmca/root.cer>>sddc-manager.example.com.pem

8. On SDDC manager SSH session take backup of the older key and certificate

   cp /etc/ssl/private/vcf_https.key /etc/ssl/private/old_vcf_https.key
   cp /etc/ssl/certs/vcf_https.crt /etc/ssl/certs/old_vcf_https.crt

9. Upload sddc-manager.example.com.pem chain to /home/vcf
10. Replace the key and certificate with new one that was generated

    rm /etc/ssl/certs/vcf_https.crt
    mv /home/vcf/sddc-manager.example.com.pem /etc/ssl/certs/vcf_https.crt
    mv /home/vcf/sddc-manager.example.com.key /etc/ssl/private/vcf_https.key

11. Assign permissions to crt and key file

    chmod 644 /etc/ssl/certs/vcf_https.crt
    chmod 640 /etc/ssl/private/vcf_https.key

12. Restart nginx service 

    nginx -t && systemctl reload nginx

Additional Information

• Check if VMCA root is in SDDC trust store, if not publish it using How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores
• If SDDC has expired store certificate delete it using https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/4-5/administering/certificate-management-admin/remove-old-or-unused-certificates-from-cloud-foundation-admin.html
• Delete expired trusted root certificates in vCenter if any using the KB Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)