Cannot delete CA certificate in Operations Manager 2.8.x

Products

• Operations Manager

Issue/Introduction

When trying to rotate the CA certificate and following the documentation, https://docs.pivotal.io/platform/2-8/security/pcf-infrastructure/api-cert-rotation.html#delete, the last step fails with the following:

curl "https://${OPSMGRHOST}/api/v0/certificate_authorities" -X GET -H "Authorization: Bearer ${UAA_ACCESS_TOKEN}" | jq .

{
  "certificate_authorities": [
    {
      "guid": "6edb8.....a2fb4",
      "issuer": "Pivotal",
      "created_on": "2016-05-03T14:10:01Z",
      "expires_on": "2020-04-07T08:34:51Z",
      "active": false,
      "cert_pem": "-----BEGIN CERTIFICATE-----END CERTIFICATE-----\n",
      "nats_cert_pem": "-----BEGIN CERTIFICATE---------END CERTIFICATE-----\n"
    },
    {
      "guid": "e6c57dc71ef3f67ff8a7",
      "issuer": "Pivotal",
      "created_on": "2020-01-30T09:35:49Z",
      "expires_on": "2024-01-30T09:35:49Z",
      "active": true,
      "cert_pem": "-----BEGIN CERTIFICATE----- XXX   -----END CERTIFICATE-----\n",
      "nats_cert_pem": "-----BEGIN CERTIFICATE----- XXX    -----END CERTIFICATE-----\n"
    }
  ]
}
curl "https://${OPSMGRHOST}/api/v0/certificate_authorities/6edb8.....a2fb4" -X DELETE -H "Authorization: Bearer ${UAA_ACCESS_TOKEN}"
{"errors":{"base":["undefined method `destroy!' for nil:NilClass"]}}

This article applies for the following products/versions:

• Operations Manager 2.8.2
• VMware Tanzu Application Service for VMs 2.8.3

Environment

Product Version: 2.8

Resolution

We assume that there was upgrade from Operations Manager 2.7 to 2.8. 

If you have created the cert after the upgrade to 2.8, then further investigation is required.

If you had two CA certs prior to your upgrade, then the following procedure should get you unblocked. Follow steps 1 through 3 of Method 2 linked in this article

Method 2

1. SSH into Ops Manager VM and change user to root.

2. Open the rails console:

# cd /home/tempest-web/tempest/web; RAILS_ENV='production' TEMPEST_INFRASTRUCTURE='vsphere' TEMPEST_WEB_DIR='/home/tempest-web' DATA_ROOT='/var/tempest' LOG_DIR='/var/log/opsmanager' SECRET_KEY_BASE='your-decryption-passphrase ' su tempest-web --command 'bundle exec rails console'

Replace INFRASTRUCTURE with either google, aws, azure, vsphere, or openstack.

3. Set the decryption passphrase:

irb(main):001:0> EncryptionKey.instance.passphrase = 'my-decryption-passphrase'

Replace my-decryption-passphrase with correct decryption passphrase.

Paste the following into the Rails console and press return:

CertificateRotation.create!(certificate_authorities_excluded_from_rotation: "/services/tls_ca")

Then type exit and press return.

You should now be able to delete the inactive CA Certificate. If you cannot, please send the contents of the terminal session where you performed the CertificateRotation.create! operation.

RCA

Between Ops Manager 2.7 and 2.8, additional logic to manage certificate rotations was added. Whenever generate or add a new CA, a CertificateRotation record gets created for internal Ops Manager housekeeping. Similarly, whenever you delete an inactive CA the CertificateRotation record is removed.
 

However, if you generated or added a CA in Operation Manager 2.7, then there will be no CertificateRotation record, and deletion of that inactive CA will fail in the way reported above.

Important Note: Apply Changes on all tiles after the deletion to ensure that the old CA is removed (and therefore untrusted) from all VMs.