Certificate regeneration gets violation "latest certificate authority versions with active children are not signing"

Products

• VMware Tanzu Application Service

Issue/Introduction

Operator is attempting to regenerate Tanzu certificates via procedure and gets violation - 

{"certificates":{"regenerated":[],"excluded":[],"regenerate_failed":[]},"safety_violations":[{"violation":"latest certificate authority versions with active children are not signing","certificate_names":["/p-bosh/p_spring-cloud-services-################/pxc_galera_ca","/p-bosh/p_spring-cloud-services-###########/pxc_server_ca"]}],"errors":["failed to create new inactive certificate authorities"]}u

Resolution

Refer to Operation Manager explanation for this violation -  

"violation": "latest certificate authority versions with active children are not signing",

"mitigation": "if children are not on their latest version, apply changes in Tanzu Ops Manager. If children are on their latest version, but the signing version is still wrong, it means that you are probably re-running this step and can proceed to the next step"

Find the specific certificates and deployment that safety violation is occurring against -  

{"certificates":{"regenerated":[],"excluded":[],"regenerate_failed":[]},"safety_violations":[{"violation":"latest certificate authority versions with active children are not signing","certificate_names":["/p-bosh/p_spring-cloud-services-################/pxc_galera_ca","/p-bosh/p_spring-cloud-services-###########/pxc_server_ca"]}],"errors":["failed to create new inactive certificate authorities"]}u

In the above example this is happening against Spring Cloud Service certficates pxc_server_ca and pxc_galera_ca. If there's any ambiguity about deployment then you can lookup certificates in maestro topology (certificates/maestro_topology.yml in support bundle) and verify deployments that are used by certificate.

This violation is oftentimes caused by rotation having been run against service tile but not the service instances.

The solution is to run apply changes with update-all-service-instances errand selected for the deployment that consumes flagged certificate. Sometimes this errand is called upgrade-all-service-instances or deploy-all errand as the naming will vary by tile author. Once apply changes and errand has been run then retry regenerate operation.