CNF termination fails with Pods in terminating state and Helm stuck in uninstalling status

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Cloud Native Network Function (CNF) termination fails in the VMware Telco Cloud Automation (TCA) UI.
• Certain Pods remain stuck in a terminating state on the cluster.
• Executing helm list -A --all displays the target release in an uninstalling state rather than completing the process.
• The TCA UI eventually marks the terminate operation as "Failed" or "Error" after timeout.
• The tca-helm-service logs display the following timeout error:
  unable to uninstall helm release asynchronously, err: CNF135029: 1 error occurred: * timed out waiting for the condition

Environment

TCA 3.2

Cause

1. This issue occurs due to an incorrect operational sequence within the tca-helm-service during the Helm uninstallation workflow.
2. TCA deletes the image pull secret (e.g., Harbor docker-registry secret) prior to executing the Helm pre-delete hooks. When Helm attempts to run the pre-delete hook Job, the cluster fails to pull the required container image because the authentication secret has already been removed. The hook Job hangs indefinitely, causing the tca-helm-service to reach its default  HELM_SERVICE_UNINSTALL_TIMEOUT limit, which cascades into a CNF termination failure.

Resolution

This issue is scheduled to be resolved in VMware Telco Cloud Platform (TCP) 5.2.

Workaround:
To resolve the hung state and complete the uninstallation, manually bypass the failing Helm pre-delete hooks using the Kubernetes CLI, and then retry the operation in the TCA UI.

1. Download the kubeconfig file of the target TKG cluster.
2. Export the kubeconfig to your terminal session:
   export KUBECONFIG=/path/to/downloaded/kubeconfig
3. Identify the release name and namespace of the hung CNF by listing all Helm releases:
   helm list -A --all
4. Manually uninstall the release using the --no-hooks flag. This command skips the pre-delete job that is failing to pull its image, allowing Helm to clean up the release state:
   helm uninstall <release-name> -n <namespace> --no-hooks
5. Wait for the TCA UI to mark the initial terminate operation as Failed or Error.
6. From the TCA UI, select Retry (or Reset) for the Terminate operation.
7. Once the UI state clears, proceed to delete the CNF from the TCA UI or reinstantiate it as required.