Split Child Certificates During Single CA Rotation

Products

• VMware Tanzu Application Service

Issue/Introduction

During a Single CA Rotation and regenerating of child certs, both 'maestro regenerate ca' and 'maestro update-transitional signing' should generate new leaf versions.

In the event that a customer manually executes credhub commands to regenerate the CA and move the transitional flag or runs maestro garbage-collect leafs that inadvertently deleted the leaf versions, child certs (in this example, /telemetry-agent-cert) may fail to regenerate and continue to be signed by the old CA cert.

You can confirm by looking at the maestro topology report:

NEW: /telemetry-ca-cert

- name: /telemetry-ca-cert
      certificate_id: 00000000-0000-0000-0000-000000000000
      signed_by: /telemetry-ca-cert
      versions:
        - version_id: new11111-1111-1111-1111-111111111111
          active: true
          signing: true
          certificate_authority: true
          generated: true
          valid_until: 2026-05-04T22:19:42Z

OLD: /telemetry-ca-cert       
        - version_id: old22222-2222-2222-2222-222222222222
          signing: true
          transitional: true
          certificate_authority: true
          generated: true
          valid_until: 2025-08-02T16:14:02Z
          
 signs:
        - name: /telemetry-agent-cert
          certificate_id: 744d0613-e0b5-439c-8cfb-3f7254079245
          signed_by: /telemetry-ca-cert
          versions:
            - version_id: 3f634b12-ecda-4284-88e6-f9c4bf0f4982
              active: true
              signed_by_version: old22222-2222-2222-2222-222222222222

         - name: /telemetry-centralizer-cert
          certificate_id: 8e1d8d96-0252-4d18-8786-339db353ba97
          signed_by: /telemetry-ca-cert
          versions:
            - version_id: fb73b99c-76f3-45c3-91a1-874c2b28161f
              active: true
              signed_by_version: new11111-1111-1111-1111-111111111111  

 

Cause

This can cause safety violations during the rotation process:

safety_violations:
    - violation: active child certificate does not trust the certificate authority that will regenerate it
      certificate_names:
        - /telemetry-agent-cert
        - /telemetry-centralizer-cert
error: safety constraints violated

Resolution

Switch the transitional flag back from old cert (22222-2222-2222-2222-222222222222) to the new cert (11111-1111-1111-1111-111111111111):

credhub curl -p "/api/v1/certificates?name=/telemetry-ca-cert"

{
  "certificates": [
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "versions": [
        {
          "id": "new11111-1111-1111-1111-111111111111",
          "transitional": false,
          ...
        },
        {
          "id": "old22222-2222-2222-2222-222222222222",
          "transitional": true,
          ...
        }
      ]
    }
  ]
}

credhub curl -p /api/v1/certificates/00000000-0000-0000-0000-000000000000/update_transitional_version -d '{"version": "11111-1111-1111-1111-111111111111"}' -X PUT

Confirm the two CA certs transitional flags have been swapped:

$ credhub curl -p "/api/v1/data?name=/example-ca&current=true"

credhub curl -p "/api/v1/certificates?name=/telemetry-ca-cert"

{
  "certificates": [
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "versions": [
        {
        "id": "new11111-1111-1111-1111-111111111111",
          "transitional": true,
          ...
        },
        {
        "id": "old22222-2222-2222-2222-222222222222",
          "transitional": false,
          ...
        }
      ]
    }
  ]
}

Reference:- https://github.com/pivotal/credhub-release/blob/main/docs/ca-rotation.md#step-2-moving-the-transitional-flag

Regenerate new leaf certs:

maestro regenerate leaf --signed-by /telemetry-ca-cert

Run an apply change on the entire foundation. Once that is complete, please run the below command and proceed with the rotation:

maestro update-transitional signing --name "/telemetry-ca-cert"