maestro command fails with "safety violation: active certificate version is not the latest non transitional version"

Products

• VMware Tanzu Application Service

Issue/Introduction

This safety violation error might occur when attempting to rotate service leafs when running command:

$ maestro update-transitional remove --name /services/tls_ca

safety_violations: 

16:25:50    - violation: active certificate version is not the latest non transitional version 
...

This procedure is performed to rotate leaf certificates for service tiles: https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-operations-manager/3-0/tanzu-ops-manager/security-pcf-infrastructure-advanced-certificate-rotation.html#services-rotation

Cause

Certain service tiles have leaf certificates that have migrated the signing CA for leafs. Credhub holds onto reference to the leaf cert with old signing CA and this leads to safety violation error.

Resolution

It is advised to have Broadcom support to review the certificate state and the environment prior to running the workaround command. To open a support ticket go to Broadcom Support 

The workaround to this issue is to perform a garbage-collect prior to running update-transitional flag command. 

Run command:

$ maestro garbage-collect leaf

This will clean up any stale certificates that have had signing CA migrated. Once garbage-collected, continue with certificate rotation procedure at step of failure.