Intermittent Authentication Issues Using Kubeconfig

Products

• VMware Telco Cloud Automation

Issue/Introduction

Authentication using downloaded kubeconfig files fails intermittently. This affects both standard and recovery kubeconfig downloads, resulting in inability to execute Kubernetes operations (e.g., NF management).

TCA logs show RBAC-related errors and failed queries to TCA-CP APIs:

Sample log entries (/opt/vmware/logs/web.log):

No or null result for running a postgres query as function rbac
Error while querying inventory data from TCA-CP [{"code":"E_INTERNAL_ERROR","message":"Invalid credentials"}]

TCA Proxy log example:

K8s request unauthorized! Reason: Cannot get NF.
java.util.NoSuchElementException: No rows found with token key

Environment

2.3

Cause

The problem appears to be linked to expired or missing metadata for Kubernetes service accounts used for authentication and terminal access. These service accounts may not have the tca.vmware.com/lastused annotation, leading to authorization failures in both kubectl access and TCA terminal access.

 

In some instances, the issue is temporarily resolved by rebooting TCA-CP nodes (e.g., tcacp01.example.com.), but this does not offer a permanent fix.

Resolution

Workaround

To restore kubectl access:
    1.    Restart the proxy service on the affected TCA-CP node.
    2.    Ensure terminal access to the CNF is functioning.
    3.    Run the following command on the VIM where the CNF instance is located to patch the service account:

kubectl annotate -n tca-rbac serviceaccount --all --overwrite tca.vmware.com/lastused='4098946816000'

    4.    This patch ensures long-term validity of the service account and should be applied once per CNF instance.

Note: Restarting before applying the patch is mandatory. Applying the patch first will not resolve the issue.

Validation :

After applying the workaround:

1. Re-download the kubeconfig and verify successful kubectl authentication.
2. Check TCA terminal access to ensure RBAC token metadata is correctly restored.

 

---