Error: "CNF130003 HelmTermination Failed. Reason: Internal Service Error Helm Termination Failed. HelmTermination task timed out"
search cancel

Error: "CNF130003 HelmTermination Failed. Reason: Internal Service Error Helm Termination Failed. HelmTermination task timed out"

book

Article ID: 418989

calendar_today

Updated On:

Products

VMware Telco Cloud Automation VMware vSphere ESXi VMware Telco Cloud Platform

Issue/Introduction

  • TCA-CP UI reports the below message.
    CNF130003    Helm Termination Failed.   Reason: Internal Server Error            Helm termination failed: 0/2 pod(s) satisfied condition 'DELETED': timed out
    CNF130003    Helm Termination Failed.   Reason: Internal Server Error            Helm Termination Failed Release 'cnf-deployment-xxx' not found


  • The deletion failure also indicates the below message : 
    uninstall: Failed to delete release: [unable to build kubernetes objects for delete: [resource mapping not found for name: "XXXXXXXXX" namespace: "XXXXXXXXX" from "": no matches for kind "XXXXXXXXX" in version "XXXXXXXXX" ensure CRDs are installed first, resource mapping not found for name: "XXXXXXXXX" namespace: "XXXXXXXXX" from "": no matches for kind "XXXXXXXXX" in version "XXXXXXXXX" ensure CRDs are installed first]]

Environment

  • VMware Telco Cloud Automation 2.x, 3.2, 3.3.

Cause

  • Required Custom Resource Definitions (CRDs) are cleared or removed before the Helm release termination completes. This prevents Helm from mapping existing cluster resources to their definitions during the deletion process.
  • Timeout failures can often be attributed to a pod getting stuck during the uninstalling phase, causing the CNF terminate operation to time out. 
  • While the UI reports the task as timing out, the original termination task successfully completed the uninstallation.
  • Attempted retries of the terminate task will fail because the release name in the helm history can no longer be found.

Resolution

Fixed in Telco Cloud Automation 3.4 and higher. See Download Broadcom Products and Software to download this release. For defects and enhancements, subscribe to this article (Reference: KB 275360) to be updated on fix status.

Workaround

  1. Allow the Terminate task to time out (approximately one hour).
  2. Navigate to Inventory > Network Function.
  3. Select the Options (three dots) icon next to the impacted network function.
  4. Click Reset State and confirm with OK.
  5. Wait for the status to show Completed.
  6. Select the Options icon again and click Delete.
  7. Perform any CNF-specific cleanup to ensure all CNF artifacts have been removed. 

If the above does not resolve the issue Contact Broadcom Support.

Additional Information

Any failures to instantiate the CNF following the deletion of the CNF will be directly attributed to CNF-related artifacts still existing. 

If CNF Instantiation failure is encountered run through the following:

  1. Click the Options (three dots) icon for the desired network function and select Rollback and click OK to confirm.
  2. Once complete, click the Options (three dots) icon for the desired network function, and select Delete.
  3. Engage the vendor as needed to fully clean up the CNF before attempting to instantiate the CNF again.

    For any further assistance Contact Broadcom Support.