Bosh-health-check interfering with non-configurable certificate rotation

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When performing the rotation of non-configurable leaf certificates as described in the documentation , one can encounter a "failed to regenerate leaf certificates" error after calling the Ops Manager API /api/v0/certificate_authorities/active/regenerate endpoint. The error may present itself in the following way:

{
  "certificates": {
    "regenerated": [],
    "excluded": [],
    "regenerate_failed": []
  },
  "safety_violations": [
    {
      "violation": "there is more than one signing version of a certificate authority",
      "certificate_names": [
        "/opsmgr/bosh_dns/tls_ca"
      ]
    }
  ],
  "errors": [
    "failed to regenerate leaf certificates"
  ]
} 

The output can prove itself to be misleading since we are presented with a "there is more than one signing version of a certificate authority" safety violation which is well documented in our official documentation. However, the violation can be misleading if the following are true:

1. You are in the middle of a certificate rotation and you have generated a new version of /opsmgr/bosh_dns/tls_ca certificate or any of its leafs, but have not yet ran Apply Changes which propagates the new version of a certificate to the deployments that use it.
2. Healthwatch is performing a health check.

If you attempt to call the Ops Manager API /api/v0/certificate_authorities/active/regenerate endpoint when the above are true, this results in a race condition.

Resolution

The app bosh-health-check, which is part of Healthwatch product, performs the following task every 10 minutes:

1. Create a deployment bosh-health
2. Create a VM bosh-health-check
3. Create a container on the above VM
4. Delete the above container
5. Delete a VM bosh-health-check
6. Delete a deployment bosh-health

The actual health check takes about 30 seconds. Once BOSH creates the deployment bosh-health, it will use the newest version of /opsmgr/bosh_dns/tls_ca certificate and proceed with performing the health check. If you execute maestro topology --name /opsmgr/bosh_dns/tls_ca when health check is in progress, the output will be the following:

topology:
  - name: /opsmgr/bosh_dns/tls_ca
    certificate_id: 51e5a00a-8678-433f-8691-a43e2829765f
    signed_by: /opsmgr/bosh_dns/tls_ca
    versions:
      - version_id: 17d262db-a3b5-4e16-9dcc-5cb83fded06d
        active: true              # NEW version is active
        deployment_names:
          - bosh-health           # Only the health check deployment uses the new version
        signing: true
        certificate_authority: true
        generated: true
        valid_until: 2027-10-10T17:21:39Z
      - version_id: 88cccf62-e915-4424-b49e-bb7a5eb2b055
        active: true              # OLD version is ALSO active (Safety Violation Trigger)
        deployment_names:         # All other deployments still rely on the old version
          - appMetrics-a0a1ece9204ce16d4b76
          - cf-0aeaa5a9fd7974c9262d
          - p-healthwatch2-d636ef07d4d55c5df71a
          - p-healthwatch2-pas-exporter-724b029cf82039f20a46
        signing: true
        transitional: true
        certificate_authority: true
        valid_until: 2026-11-27T09:11:44Z

The above output states that there are now two versions of /opsmgr/bosh_dns/tls_ca certificate that are active but only the bosh-health deployment uses the newer version of the cert. If you execute maestro topology --name /opsmgr/bosh_dns/tls_ca when health check is complete, the bosh-health deployment will not exist anymore and the output will be the following:

topology:
  - name: /opsmgr/bosh_dns/tls_ca
    certificate_id: 51e5a00a-8678-433f-8691-a43e2829765f
    signed_by: /opsmgr/bosh_dns/tls_ca
    versions:
      - version_id: 17d262db-a3b5-4e16-9dcc-5cb83fded06d
        certificate_authority: true  # Notice: the new version is no longer active
        generated: true
        valid_until: 2027-10-10T17:21:39Z
      - version_id: 88cccf62-e915-4424-b49e-bb7a5eb2b055
        active: true                 # Only the old version remains active
        deployment_names:            # All deployments are back to a single shared CA version
          - appMetrics-a0a1ece9204ce16d4b76
          - cf-0aeaa5a9fd7974c9262d
          - p-healthwatch2-d636ef07d4d55c5df71a
          - p-healthwatch2-pas-exporter-724b029cf82039f20a46
        signing: true
        transitional: true
        certificate_authority: true
        valid_until: 2026-11-27T09:11:44Z 

Conclusion

This issue represents a transient race condition caused by Healthwatch automated validation cycle. Because the health check deployment is short-lived (typically lasting 30 seconds within a 10-minute window), the conflict is self-resolving.

Recommended Action:

1. Wait: Allow approximately 5–10 minutes for the Healthwatch health check to complete and for BOSH to automatically delete the temporary deployment.

2. Retry: Re-run the call to the Ops Manager API /api/v0/certificate_authorities/active/regenerate endpoint.

Once the temporary deployment is gone, the safety violation will clear, as only one version of the CA will be active.

After rotation has been complete, Apply Changes should propagate the newer version of the certificate in all the deployments that use it and the older version of the cert will be deactivated.