SDDC Manager UI inaccessible or Regenerate button greyed out due to expired certificate in VMware Cloud Foundation

Products

VMware Cloud Foundation

Issue/Introduction

SDDC manager UI is down or If you are unable to generate a csr from the SDDC UI
Using CA certificate but certificate is expired when checked using below command
```
/opt/vmware/sddc-support/sos --certificate-health
```
If unable to generate certificate from CA server, momentarily to restore production we can replace SDDC certificates to VMCA signed.

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x

Cause

This issue occurs when the SDDC Manager's machine SSL certificate (vcf_https.crt) has already expired. Because the SDDC Manager UI and automated rotation services rely on a valid secure connection to function, the "Regenerate" button becomes disabled once the certificate is invalid. This creates a circular dependency that requires manual replacement via the CLI to restore service

Resolution

Note: It is strongly recommended to take a snapshot of the SDDC Manager VM and Management vCenter Server before proceeding with the steps below.

Script Method:

For the scripted method download the script attached to the bottom of this article and run it as root user on the SDDC Manager appliance in the /home/vcf directory:

SSH to SDDC Manager using vcf account and su to root.
Transfer the script to /home/vcf directory using winscp.
Run script using:

./renew_sddc_cert_with_vmca.sh

After running this script it might fail with the error :

/bin/bash^M: bad interpreter: No such file or directory

We can resolve it by executing the below command & rerun the script.

sed -i -e 's/\r$//' renew_sddc_cert_with_vmca.sh

Manual Method:

1.SSH to SDDC Manager with vcf and su to root.

2.Generate the CSR on the SDDC Manager by executing the command below. It's recommended to run this command from the /home/vcf directory:

openssl req -new -newkey rsa: 2048 -nodes -keyout sddc-manager. example.com.key -out sddc-manager.example.com.csr

3.SSH to the management vCenter and create a directory for these certificates.

mkdir /tmp/certs

4.Move the sddc-manager.example.com.csr to the above path from SDDC manager to vCenter either using WINSCP or VI

5.On the Management vCenter Server, create a new configuration file named cert.cfg using a text editor:

mkdir /tmp/certs

6.Copy and paste the configuration block below into the file.

CRITICAL: You must customize the placeholder values within the [ v3_req ] section to match your specific environment before saving:

Replace <Management_vcenter_FQDN> with the FQDN of your vCenter.
Replace <SDDC ip_address> with the IP of your SDDC Manager.
Replace <SDDC FQDN> with the FQDN of your SDDC Manager.

[ req ]
req_extensions = v3_req

[ v3_req ]
extendedKeyUsage = serverAuth, clientAuth
authorityKeyIdentifier=keyid, issuer
authorityInfoAccess = caIssuers; URI:https://mgmt-vcenter. example.com/afd/vecs/ca
subjectAltName=IP :<SDDC ip_address>, DNS :< SDDC FQDN>

Save and close the file (Press Esc, type :wq, and press Enter).

7.Generate certificate using the cfg file and CSR.

openssl x509 -req -days 3650 -in sddc-manager.example.com.csr -out sddc-manager.example.com.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile cert.cfg

8.Create chain containing vCenter VMCA root and the certificate generated using below command.

cat sddc-manager.example.com.crt>>sddc-manager.example.com.pem
cat /var/lib/vmware/vmca/root.cer>>sddc-manager.example.com.pem

9.On SDDC manager SSH session take backup of the older key and certificate

cp /etc/ssl/private/vcf_https.key /etc/ssl/private/old_vcf_https.key
cp /etc/ssl/certs/vcf_https.crt /etc/ssl/certs/old_vcf_https.crt

10.Upload sddc-manager.example.com.pem chain to /home/vcf.

11.Replace the key and certificate with new one that was generated

rm /etc/ssl/certs/vcf_https.crt
mv /home/vcf/sddc-manager.example.com.pem /etc/ssl/certs/vcf_https.crt
mv /home/vcf/sddc-manager.example.com.key /etc/ssl/private/vcf_https.key

12.Assign permissions to crt and key file

chmod 644 /etc/ssl/certs/vcf_https.crt
chmod 640 /etc/ssl/private/vcf_https.key

13.Restart nginx service

nginx -t && systemctl reload nginx

Additional Information

Check if VMCA root is in SDDC trust store, if not publish it using How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores
If SDDC has expired store certificate delete it using https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/4-5/administering/certificate-management-admin/remove-old-or-unused-certificates-from-cloud-foundation-admin.html
Delete expired trusted root certificates in vCenter if any using the KB Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)
Prerequisite check for certificate health. Using the VCF Diagnostic Tool for vSphere (VDT)

Attachments

renew_sddc_cert_with_vmca.sh get_app