VCF Service Manager: vksm service installation fails after upgrade with "ConfigurationHandler ... already exists" error

Products

VCF Automation

Issue/Introduction

After upgrading VCF Automation, installation of the vksm service (or another VCF Service Manager-managed service) fails with an error similar to:

Installation error(s); Details: [Failed to submit resources: failed to validate create for
services.vcfa.broadcom.com/v2/ConfigurationHandler/prelude/<uuid>: configurationhandlers.services.vcfa.broadcom.com
"<uuid>" already exists]

This may be accompanied by a controller log entry such as:

"msg":"Installation failed","controller":"service-rde-v2", ... ,"serviceName":"auto-attach.vksm.broadcom.com",
"errors":["Failed to submit resources: failed to validate create for services.vcfa.broadcom.com/v2/ConfigurationHandler/prelude/<uuid>: ... already exists"]

This symptom can occur on its own, without other unrelated installation failures.

Note:

This issue is isolated to environments where the prior installation never reached a successful completion state (e.g., due to an unhealthy Supervisor or a stuck uninstallation process).
Environments where the service was successfully installed but subsequently became unhealthy will not encounter this collision.

Environment

VCF Automation 9.1.1.0

Cause

Prior to the upgrade, the service's previous version never completed installation successfully - for example, because it was installed on top of a non-functional/unhealthy Supervisor, or because a prior uninstall attempt got stuck and left some of its resource custom resources (CRs) behind.
When the upgrade runs, the new version's installation attempts to create its own ConfigurationHandler resources rather than taking over the leftover ones from the incomplete previous installation. Because the leftover resources still exist, the new version's create request collides with them, producing the "already exists" error and failing the installation.
This is expected to occur only when the previous installation of the service was left in a broken/incomplete state at the time of upgrade. If the previous installation completed successfully - even if the service later became unhealthy - this issue should not occur.

Resolution

Workaround:

Clear the orphaned resources from the environment before attempting the operation again:

NOTE: Only clean up resources from stale installations (non-Healthy status, mismatched version/service-id labels). Deleting active resources will cause an outage.

Identify the affected service's `refName` from the error message (e.g. `auto-attach.vksm.broadcom.com`) and its VCFA CR namespace (the `prelude` namespace in the reported error).

List the resources labeled for that service, across the relevant resource types (`configurationhandler`, `supervisorservice`, `vcfaserviceaccount`, `role`), by refName label:

LABEL="services.vcfa.broadcom.com/refName=<service-refName>"
for RESOURCE_TYPE in configurationhandler supervisorservice vcfaserviceaccount role; do
kubectl get "${RESOURCE_TYPE}" -l "${LABEL}" -A -o json | jq -c '.items[]'
done

Inspect each object's `.metadata.namespace`, `.metadata.name`, `.metadata.labels["services.vcfa.broadcom.com/service-id"]`, `.metadata.labels["services.vcfa.broadcom.com/version"]`, and `.status.status`.

Compare the `service-id`/`version` labels and status of each resource against the version currently being installed. Resources carrying an older `version` label (or a `service-id` that no longer matches the active install), and sitting in a non-Healthy status (e.g. `Busy`, `Unhealthy`), are leftovers from the incomplete previous installation - such as a `ConfigurationHandler` stuck in `Busy` because its associated `SupervisorService` never became `Healthy`.
Before deleting anything manually, first try to uninstall the stale service (e.g. `auto-attach`) through the normal, supported uninstall flow. This gives the owning controller a chance to clean up its resources properly.
If the uninstall gets stuck (the service remains stuck in a removal/uninstalling state and its resources are not cleaned up), then fall back to manually deleting the stale leftover resources identified in step 3:
```
kubectl delete <kind>.services.vcfa.broadcom.com <name> -n prelude
```
Because these resources carry finalizers (e.g. 'services.vcfa.broadcom.com/vcfa-service', 'services.vcfa.broadcom.com/configurationhandler.callback'), the delete may hang if the owning controller cannot process the finalizer on its own (for example because the Supervisor is still unreachable). Only in that case, patch out the finalizer as a last resort:
```
kubectl patch <kind>.services.vcfa.broadcom.com <name> -n prelude --type=merge -p '{"metadata":{"finalizers":[]}}'
```
Once the stale resources are removed, retry the service installation/upgrade. The installation should now be able to create its own resources without colliding with leftovers from the prior incomplete install.