Apply Changes fail for a purged Service Instance during CA cert rotation for TAS platform.

Products

• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Application Service

Issue/Introduction

You are performing a certificate rotation procedure on your TAS/TPCF platform and Apply Changes fails on a Service Instance that was previously deleted.

At some point before attempting the cert rotation activities, there was a  cf purge command issued to delete the Service Instance referenced above. 

There was not run an Apply-Changes in between time of 'cf purging' the service instance and performing the 'cert rotation'. 

You are on Step 4 on the cert rotation procedure instructions, and Apply Changes fails with errors pertaining to the previously purged Service Instance.  

ubuntu@opsmgr-tas-ABC-cloud-<EXAMPLE>-com:~$ curl "https://opsmgr.tas-ABC.cloud.<EXAMPLE>.com/api/v0/certificate_authorities/generate" -X POST -H "Authorization: Bearer $token" -H "Content-Type: application/json" -d '{}' -i -k
HTTP/1.1 422 Unprocessable Content
{"certificates":{"regenerated":[],"excluded":[],"regenerate_failed":[]},"safety_violations":[{"violation":"active child certificate version is not the latest non transitional version or more than one active version exists","certificate_names":["/telemetry-agent-cert","/telemetry-centralizer-cert"]}],"errors":["failed to create new inactive certificate authorities"]

Cause

There was not an Apply Changes performed after purging the Service Instance and before starting the cert rotation process. Therefore, Bosh is not aware of the deleted/purged Service Instance, thus causing this issue.

This can be confirmed by running below commands:

# 1.)  Look for the service instance that is referenced in your Apply Change error.  If you can find the service instance name in this output, this means Bosh still thinks the deployment/SI is active. See example below:

# Command:
bosh vms
------------------------
# Output:
Deployment 'service-instance_1234-5678-910'

Instance  Process State  AZ  IPs  VM CID  VM Type  Active  Stemcell  VM Created At  Uptime  Load           CPU    CPU   CPU  CPU   Memory  Swap   System      Ephemeral   Persistent  
                                                                                            (1m, 5m, 15m)  Total  User  Sys  Wait  Usage   Usage  Disk Usage  Disk Usage  Disk Usage  

0 vms

 

# 2.) Look for the service instance using cf curl, example below:

# Command:
cf curl /v3/service_instances/service-instance_1234-5678-910
------------
# Output:
{"errors":[{"detail":"Service instance not found","title":"CF-ResourceNotFound","code":10010}]}

 

The above confirms cf is aware of the purged service instance, and Bosh is not.

Resolution

If the service instance is not required/desired to exist, first delete from Bosh side using below command:

bosh -d service-instance_1234-5678-910 delete-deployment

# replace the service-instance name with the name of your service instance (that appears in the Apply Change error).

If wanting to re-establish the service instance as part of the deployment, use below commands:

bosh -d service-instance_1234-5678-910 manifest > service-instance_1234-5678-910.yml

bosh -d service-instance_1234-5678-910 deploy service-instance_1234-5678-910.yml

# replace the service-instance name with the name of your service instance (that appears in the Apply Change error).

Performing above actions should unblock the issue, then can run Apply Changes again.

Continue Rotation steps.