VMware Tanzu GemFire [VMs] Maestro Certificate Rotation Issue

Products

VMware Tanzu Gemfire

Issue/Introduction

Who is affected?

All VMware Tanzu GemFire [VMs] customers using maestro to rotate their /services/tls_ca

What is the problem?

When using the maestro tool to rotate /services/tls_ca there is a possibility that users could encounter the following error message:

safety_violations:
    - violation: there is more than one signing version of a certificate authority
      certificate_names:
        - /services/tls_ca
error: safety constraints violated

Resolution

A bit of background on Maestro:

CredHub Maestro is a command-line interface (CLI) that facilitates rotations of certificates. Using the tool, you can:

Determine if any of your CredHub certificates are expiring soon
Rotate CredHub certificates
Clean up inactive certificate versions so that CredHub does not run out of disk space

For more information about setting up and using CredHub Maestro, see Getting Started with CredHub Maestro.

What’s the root cause?

If a new version of /services/tls_ca is on top of the default generated version. There is a potential to have 2 versions of /services/tls_ca with the signing set to true. This causes the maestro tool to stop generating a new certificate until one of the signing versions is removed.

Example: Here’s the output of the command `maestro topology`

topology:
    - name: /services/tls_ca
      certificate_id: 1b84c09d-8a11-452d-bedd-8f6b78b679d9
      signed_by: /services/tls_ca
      versions:
        - version_id: c9a46190-2f2f-45d9-9003-e44c80d9b800
          signing: true
          certificate_authority: true
          generated: false
          valid_until: 2022-02-24T21:15:52Z
        - version_id: b3a78eed-c818-4690-9231-dcaf35179c9c
          signing: true
          certificate_authority: true
          generated: true
          valid_until: 2026-02-23T21:46:43Z
      signs:
        - name: /p-bosh/service-instance_c43ff3dd-fc8b-4cf6-a50c-d48f8db03094/gemfire-locator-certificate
          certificate_id: a1387109-2f09-4f8b-afef-219dfff217c3
          signed_by: /services/tls_ca
          versions:
            - version_id: 77a3a5cc-0bc9-418f-9172-0e76095c1510
              active: true
              signed_by_version: c9a46190-2f2f-45d9-9003-e44c80d9b800
              deployment_names:
                - service-instance_c43ff3dd-fc8b-4cf6-a50c-d48f8db03094
              generated: true
              valid_until: 2022-02-25T02:41:05Z
        - name: /p-bosh/service-instance_c43ff3dd-fc8b-4cf6-a50c-d48f8db03094/gemfire-server-certificate
          certificate_id: c27dd46d-4f5f-4fc3-ab88-54e5d2e598df
          signed_by: /services/tls_ca
          versions:
            - version_id: 677eac15-7701-4364-bf81-451f04b99d55
              active: true
              signed_by_version: c9a46190-2f2f-45d9-9003-e44c80d9b800
              deployment_names:
                - service-instance_c43ff3dd-fc8b-4cf6-a50c-d48f8db03094
              generated: true
              valid_until: 2022-02-25T02:41:04Z
        - name: /services/tls_leaf
          certificate_id: 90f3eda9-9b22-409f-8df7-f054415cf151
          signed_by: /services/tls_ca
          versions:
            - version_id: 5dffc99e-988a-413c-b9d9-af75520d89ef
              active: true
              signed_by_version: b3a78eed-c818-4690-9231-dcaf35179c9c
              deployment_names:
                - cf-af156c52a71c3cc2bfc9
              certificate_authority: true
              generated: true
              valid_until: 2022-02-24T21:57:44Z

In the above topology we can see that both the versions c9a46190-2f2f-45d9-9003-e44c80d9b800 and b3a78eed-c818-4690-9231-dcaf35179c9c have signing set to true i.e signing: true.

What is the workaround?

VMware Tanzu Maestro team is aware of the issue and working on a resolution. Until a workaround is proposed, please rotate certs manually using the CredHub.