Replacing a NSX-T or SDDC Manager certificate through fails due to not being able to decode the rootca.crt

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• In the /var/log/vmware/vcf/common-svcs/vcf-commonsvcs.log the SDDC Manager certificate replacement fails with below error:

2021-05-10T11:32:05.084+0000 INFO  [common,250eda9f59a257cb,f1f8] [c.v.e.s.a.u.NginxCertUtilityImpl,http-nio-127.0.0.1-7100-exec-15] Reloading NGINX server ...
2021-05-10T11:32:05.136+0000 ERROR [common,250eda9f59a257cb,f1f8] [c.v.e.s.c.util.LocalProcessService,http-nio-127.0.0.1-7100-exec-15] Local Command Failed with exit value 1.
Output Logs :
Error Logs are stored at LocalProcess ERROR: 2021-05-10 11:32:05 - + set -e
LocalProcess ERROR: 2021-05-10 11:32:05 - + nginx -t
LocalProcess ERROR: 2021-05-10 11:32:05 - nginx: [emerg] cannot load certificate "/etc/ssl/certs/vcf_https.crt": PEM_read_bio_X509() failed
LocalProcess ERROR: 2021-05-10 11:32:05 - nginx: configuration file /etc/nginx/nginx.conf test failed
LocalProcess ERROR: 2021-05-10 11:32:05 - + exit 1
2021-05-10T11:32:05.136+0000 ERROR [common,250eda9f59a257cb,f1f8] [c.v.e.s.a.u.NginxCertUtilityImpl,http-nio-127.0.0.1-7100-exec-15] Nginx reload script failed to execute

• Attempts to replace the NSX-T certificate fails and reports the following in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log:

 

2021-05-10T11:32:06.728+0000 ERROR [vcf_om,d5e87ee2f1f14250,261b] [c.v.v.c.n.NsxTManagerCertificatePlugin,om-exec-19] 500 : [{"error_code": 36233, "error_message": "Error updating http service certificate.", "module_name": "node-services"}]
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : [{"error_code": 36233, "error_message": "Error updating http service certificate.", "module_name": "node-services"}]

Also,if someone tries to decode rootca.crt,below error is coming:
openssl x509 -in /opt/vmware/vcf/operationsmanager/certificate/<domain_name>/rootca.crt -noout -text
unable to load certificate

• Trying to decode rootca.crt with the command below fails with "Unable to load certificate":
  openssl x509 -in /opt/vmware/vcf/operationsmanager/certificate/<domain_name>/rootca.crt -noout -text
   

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

 

Environment

VMware Cloud Foundation 3.10.0.0
VMware Cloud Foundation 3.10.x
VMware Cloud Foundation 3.10.1.2
VMware Cloud Foundation 3.10.1.1
VMware Cloud Foundation 4.2

Cause

The OpenSSL 1.0.2 fails to load the certificate in certain cases when the certificate content is of length in the order of 253 multiples. Ref: https://github.com/openssl/openssl/issues/9187

Resolution

This is a known issue affecting VCF 4.1.2, VCF 3.10.1.2 and older releases. This is fixed in the later releases of VCF.

Workaround:

1. Open an SSH session to the SDDC Manager VM and switch to root user:

su -

2. Take the backup of /opt/vmware/vcf/operationsmanager/certificate/<domain_name>/rootca.crt, where domain_name is the workload domain for the component that failed the certficate replacement.
3. Execute below command to format the line length to 64 characters, ensuring to specify the domain:

fold --spaces --width=64 /opt/vmware/vcf/operationsmanager/certificate/<domain_name>/rootca.crt  > /tmp/tmp.crt

4. Replace the rootca.crt with the newly formatted certifcate.

mv /tmp/tmp.crt /opt/vmware/vcf/operationsmanager/certificate/<domain_name>/rootca.crt

5. Re-trigger/Retry the workflow.