Certificate regeneration gets violation "latest certificate authority versions with active children are not signing"

Products

• VMware Tanzu Application Service

Issue/Introduction

Operator is attempting to regenerate Tanzu certificates via the standard rotation procedure  

{
  "certificates": {
    "regenerated": [],
    "excluded": [],
    "regenerate_failed": []
  },
  "safety_violations": [
    {
      "violation": "latest certificate authority versions with active children are not signing",
      "certificate_names": [
        "/p-bosh/p_spring-cloud-services-################/pxc_galera_ca",
        "/p-bosh/p_spring-cloud-services-###########/pxc_server_ca"
      ]
    }
  ],
  "errors": [
    "failed to create new inactive certificate authorities"
  ]
}

Or operator is attempting to regenerate Tanzu certificates via the advanced maestro rotation procedure  

 maestro regenerate ca --name "/telemetry-ca-cert"
safety_violations:
    - violation: certificate authorities can not have a transitional version
      certificate_names:
        - /telemetry-ca-cert
    - violation: latest certificate authority versions with active children are not signing
      certificate_names:
        - /telemetry-ca-cert
error: safety constraints violated

Resolution

Refer to Operation Manager Documentation for explanation of this violation 

https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-operations-manager/3-0/tanzu-ops-manager/security-pcf-infrastructure-troubleshoot-cert-errors.html

 

Service Instance Example

Find the specific certificates and deployment that safety violation is occurring against -  

{
  "certificates": {
    "regenerated": [],
    "excluded": [],
    "regenerate_failed": []
  },
  "safety_violations": [
    {
      "violation": "latest certificate authority versions with active children are not signing",
      "certificate_names": [
        "/p-bosh/p_spring-cloud-services-################/pxc_galera_ca",
        "/p-bosh/p_spring-cloud-services-###########/pxc_server_ca"
      ]
    }
  ],
  "errors": [
    "failed to create new inactive certificate authorities"
  ]
}

In the above example this is happening against Spring Cloud Service certficates pxc_server_ca and pxc_galera_ca. If there's any ambiguity about deployment then you can lookup certificates in maestro topology (certificates/maestro_topology.yml in support bundle) and verify deployments that are used by certificate.

This violation is oftentimes caused by rotation having been run against service tile but not the service instances.

The solution is to run apply changes with update-all-service-instances errand selected for the deployment that consumes flagged certificate. Sometimes this errand is called upgrade-all-service-instances or deploy-all errand as the naming will vary by tile author. Once apply changes and errand has been run then retry regenerate operation.

 

Maestro Root CA regenerate Example

 

 maestro regenerate ca --name "/telemetry-ca-cert"
safety_violations:
    - violation: certificate authorities can not have a transitional version
      certificate_names:
        - /telemetry-ca-cert
    - violation: latest certificate authority versions with active children are not signing
      certificate_names:
        - /telemetry-ca-cert
error: safety constraints violated

 

in this example the /telemetry-ca-cert has a new version telemetry-ca-cert that is marked transitional but not active. In this example the version 985-GUID is newer than the active version bc6-GUID.  Someone may have regenerated this certificate in the past but never completed the rotation. 

 

"maestro topology --name /telemetry-ca-cert" output

name: /telemetry-ca-cert
certificate_id: b5c-GUID
signed_by: /telemetry-ca-cert
versions:
- version_id: 985-GUID
  active: false.            <<<<<<<<<<<<<<<<<<<<<<<<<<<<
  signed_by_version: ""
  deployment_names: []
  signing: false.         <<<<<<<<<<<<<<<<<<<<<<<<<<<<
  transitional: true
  certificate_authority: true
  generated: true
  valid_until: 2026-05-12T20:09:50Z
- version_id: bc6-GUID
  active: true            <<<<<<<<<<<<<<<<<<<<<<<<<<<<
  signed_by_version: ""
  deployment_names:
  - appMetrics-GUID
  - bosh-health
  - cf-GUID
  signing: true           <<<<<<<<<<<<<<<<<<<<<<<<<<<<
  transitional: false   <<<<<<<<<<<<<<<<<<<<<<<<<<<<
  certificate_authority: true
  generated: true
  valid_until: 2026-02-18T18:35:12Z

In this case you may want to simply continue with the rotation by skipping the regenerate step.  Given the latest cert was over 6 months old the operator wants to rollback the rotation. Here is how you can rollback the rotation by first removing the latest transitional cert

1. Verify the latest cert is transitional true, signing is false, active is false, and deployment_names is empty. If either of those conditions do not match then this procedure is not for you. 
2. remove the transitional flag

   1. maestro update-transitional remove --name "/telemetry-ca-cert" --skip-safety-check

3. Run garbage collection with --dry-run flag first.  With the dry run flag it will tell you which certs it will delete.  Make sure the version IDs match the versions we want deleted in the maestro topology output.

   1. maestro gc ca --force -name "/telemetry-ca-cert" --dry-run
      
      ## validate dry run look good before running the actual garbage collection
      
      maestro gc ca --force -name "/telemetry-ca-cert"
      

4. Proceed to regenerate the root ca certificate and follow the advanced maestro rotation procedure.