CNF instantiation fails with unauthorized CNF130005

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Following an underlay network disruption (such as a physical switch upgrade or BGP drop), CNF instantiations fail globally across multiple Workload Clusters managed by the same vCenter/Management Cluster.
• The TCA Control Plane (TCA-CP) logs (postgres/tca-cp-cn/tca_sensitive.out) register the following error during the placement grant request:
  "errorCode":"CNF130005","internalMessage":"Grant request failed: Error loading Kubernetes API resources: : Unauthorized"
• Additionally, the kube-apiserver audit logs across all Workload and Management clusters show rejected requests referencing an ephemeral CNF service account globally.
  "Unable to authenticate the request" err="[invalid bearer token, serviceaccounts \"NAME\" not found]"

Environment

Telco Cloud Automation 3.4

Cause

An abrupt network termination causes a state management defect within the TCA-CP connection pool. A transient namespace-scoped ServiceAccount token (used temporarily for CNF deployment) becomes stuck in the TCA-CP memory cache due to a bad socket closure. When network connectivity is restored, TCA-CP applies this orphaned token to global polling loops for all registered Kubernetes endpoints. The target API servers evaluate the cross-pollinated token, query their local etcd datastores, and reject the requests as unauthorized because the specific service account does not exist globally.

Resolution

• Initiate a manual kubeconfig synchronization for the impacted clusters via the TCA API to force the TCA Manager to pull fresh, valid administrative context from the TKG Management Cluster.
• Reference the exact API payload and steps in Broadcom KB 345733.
• If the cache persists, log in via SSH to the affected TCA Control Plane (TCA-CP) appliance.
• Execute the following command to restart the application microservice, which forces a complete flush of the in-memory connection pools and credential caches:
  

  kubectl rollout restart deployment tca-app -n tca-cp-cn

• Wait for the newly spawned tca-app pods to reach a Running state and verify that CNF instantiation workflows execute successfully.