Apply Changes fails with "crypto/rsa: verification error" after rotating and activating a new root CA for Operations Manager

Products

• Operations Manager

Issue/Introduction

After rotating/activating a new root CA and deleting the old root CA for Operations (Ops) Manager, Apply Changes is failing with the following error:

Creating instance 'bosh/0': Waiting until instance is ready: Post https://vcap:@10.193.119.11:6868/agent: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Pivotal") Exit code 1

Environment

Product Version: 2.6

Resolution

Make sure you have successfully rotated non-configurable leaf certificates from the new root CA. You can check in the access.log if the regenerate api was successful.

From /var/log/opsmanager/access.log in Ops Manager VM, you can see a new root CA was created, then activated, regenerate API failed, then old root CA was deleted:

10.0.0.132 - - [05/Jan/2021:17:18:59 +0000] "POST /api/v0/certificate_authorities/generate HTTP/1.1" 200 2704 "-" "curl/7.47.0"
10.0.0.132 - - [05/Jan/2021:17:38:47 +0000] "POST /api/v0/certificate_authorities/560089ede69d239895ca/activate HTTP/1.1" 200 12 "-" "curl/7.47.0"
10.0.0.132 - - [05/Jan/2021:18:30:50 +0000] "POST /api/v0/certificate_authorities/active/regenerate HTTP/1.1" 422 135 "-" "Go-http-client/1.1"
10.0.0.132 - - [05/Jan/2021:18:56:51 +0000] "DELETE /api/v0/certificate_authorities/df81d4c583fb648500fa HTTP/1.1" 200 12 "-" "curl/7.47.0"

Use the regenerate API to regenerate all non-configurable certificates and apply the new root CA to your existing BOSH Director, use -v flag to get more verbosity for the http response.

curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/active/regenerate" \ -X POST \ -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \ -H "Content-Type: application/json" \ -d '{}' -v 

The API returns a successful response:

HTTP/1.1 200 OK 

For more details, please refer to Rotate non-configurable leaf certificates in the Tanzu Application Service documentation.

 

If you experience a "Safety Violation" please consult the following documentation https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/security-pcf-infrastructure-troubleshoot-cert-errors.html 

If you are unable to get past the safety violation please open a case with Tanzu support