SDDC Manager UI inaccessible or Regenerate button greyed out due to expired certificate in VMware Cloud Foundation

Products

VMware SDDC Manager / VCF Installer VMware vCenter Server

Issue/Introduction

SDDC manager UI is down or If you are unable to generate a csr from the SDDC UI
Using CA certificate but certificate is expired when checked using below command
```
/opt/vmware/sddc-support/sos --certificate-health
```
If unable to generate certificate from CA server, momentarily to restore production we can replace SDDC certificates to VMCA signed.
VCF Operations inventory sync tasks fail with error "Service or view is not available for the selected object"

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

VMware Cloud Foundation 9.x

Cause

This issue occurs when the SDDC Manager's machine SSL certificate (vcf_https.crt) has already expired. Because the SDDC Manager UI and automated rotation services rely on a valid secure connection to function, the "Regenerate" button becomes disabled once the certificate is invalid. This creates a circular dependency that requires manual replacement via the CLI to restore service

Resolution

Note: It is strongly recommended to take a snapshot of the SDDC Manager VM and Management vCenter Server before proceeding with the steps below.

Script Method

For the scripted method, download the attached script and run it as the root user on the SDDC Manager appliance within the /home/vcf directory.

SSH to the SDDC Manager using the vcf account and elevate to root:
```
su -
```
Transfer the renew_sddc_cert_with_vmca.sh script to the /home/vcf directory using WinSCP or a similar tool.
Execute the script:
```
./renew_sddc_cert_with_vmca.sh
```

Troubleshooting

If the script fails with the error /bin/bash^M: bad interpreter: No such file or directory, perform the following steps to resolve line-ending issues and re-run the script:

Execute the following command to fix the script file:
```
sed -i -e 's/\r$//' renew_sddc_cert_with_vmca.sh
```
Re-run the script:
```
./renew_sddc_cert_with_vmca.sh
```

--------------------------------------------------------------------------------------------------------------------------

Manual Method:

To manually regenerate and replace the SDDC Manager certificate, follow these steps:

SSH to the SDDC Manager as vcf and elevate to root:
```
su -
```

Generate the CSR on the SDDC Manager from the /home/vcf directory:

SSH to the Management vCenter and create a temporary directory:
```
mkdir /tmp/certs
```
Transfer the sddc-manager.example.com.csr file from the SDDC Manager to the /tmp/certs directory on the Management vCenter Server using WinSCP or a similar tool.
On the Management vCenter Server, create a new configuration file named cert.cfg in /tmp/certs:
```
vi /tmp/certs/cert.cfg
```
- Replace <Management_vcenter_FQDN> with the FQDN of the vCenter.
- Replace <SDDC ip_address> with the IP address of the SDDC Manager.
- Replace <SDDC FQDN> with the FQDN of the SDDC Manager.
  Copy and paste the following configuration block into the file. Customize the placeholders within the [ v3_req ] section before saving:
```
[ req ]
req_extensions = v3_req

[ v3_req ]
extendedKeyUsage = serverAuth, clientAuth
authorityKeyIdentifier=keyid, issuer
authorityInfoAccess = caIssuers;URI:https://<Management_vcenter_FQDN>/afd/vecs/ca
subjectAltName=IP:<SDDC ip_address>, DNS:<SDDC FQDN>
```

Generate the certificate using the cert.cfg file and the CSR:

openssl x509 -req -days 3650 -in sddc-manager.example.com.csr -out sddc-manager.example.com.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile /tmp/certs/cert.cfg

Create a certificate chain containing the vCenter VMCA root and the newly generated certificate:

On the SDDC Manager, back up the existing key and certificate:

Upload the sddc-manager.example.com.pem chain and the sddc-manager.example.com.key to /home/vcf on the SDDC Manager.

Replace the existing certificate and key with the new files:

Update the permissions for the certificate and key files:

Restart the Nginx service:
```
nginx -t && systemctl reload nginx
```

Additional Information

Check if VMCA root is in SDDC trust store, if not publish it using How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores
If SDDC has expired store certificate delete it using https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/4-5/administering/certificate-management-admin/remove-old-or-unused-certificates-from-cloud-foundation-admin.html
Delete expired trusted root certificates in vCenter if any using the KB Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)
Prerequisite check for certificate health. Using the VCF Diagnostic Tool for vSphere (VDT)

Attachments

renew_sddc_cert_with_vmca.sh get_app