Expired CA still shown in bosh env --details after Tanzu Operations Manager CA rotation until old CA is deleted

Products

• VMware Tanzu Application Service

Issue/Introduction

After rotating Tanzu Operations Manager-generated CAs and leaf certificates by using the Tanzu Operations Manager API, bosh env --details may continue to report an expired CA for one or more of the following BOSH Director certificates:

• nats.tls.ca
• director.config_server.ca_cert
• director.config_server.uaa.ca_cert

On the BOSH Director VM, the related certificate files may still contain both the old and new CA certificates concatenated in the same PEM file, for example:

• /var/vcap/jobs/director/config/nats_server_ca.pem
• /var/vcap/jobs/director/config/config_server_ca.cert
• /var/vcap/jobs/director/config/uaa_server_ca.cert

In this state, operators may conclude that the CA rotation did not succeed because bosh env --details continues to show the expiry date of the old certificate.

Environment

 

• Tanzu Operations Manager
• BOSH Director deployed by Tanzu Operations Manager
• Environments where CAs and leaf certificates were rotated using the Tanzu Operations Manager API procedure
• Situations where the final Delete old CAs step of the CA rotation procedure was not completed

 

Cause

The Tanzu Operations Manager API rotation procedure is a multi-step CA rotation workflow. After the new CA is activated and leaf certificates are regenerated, the procedure requires a final cleanup step to delete the old CA. Until that cleanup is performed, the old and new CA certificates can remain together in the same certificate bundle.

In this transitional state, bosh env --details can continue to report the old, expired CA from the certificate bundle, even though the environment may already be using the new CA for trust and communication.

Resolution

Complete the final Delete old CAs step from the Tanzu Operations Manager API CA rotation procedure. After the old CA is deleted, the Director CA bundle files contain only the active CA, and bosh env --details then reports the new certificate correctly.

After completing the deletion step, verify the result with:

 

bosh env --details

 

Also inspect the related CA files on the Director VM if needed:

 

sudo cat /var/vcap/jobs/director/config/nats_server_ca.pem
sudo cat /var/vcap/jobs/director/config/config_server_ca.cert
sudo cat /var/vcap/jobs/director/config/uaa_server_ca.cert

 

If multiple certificates were previously present, only the active CA should remain after the cleanup step.