First CNF to be onboarded loses CNF namespace privileges when a CF is removed and attempted to be instantiated

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

This fix will help users restore the remote access to the CNF namespaces when using the CNF terminal or kubectl functionality of TCA.

Symptoms:

1. CNF is terminated
2. CNF re-instantiated to the same namespace
3. Opening a terminal to the CNF
4. Execution of the command kubectl get pods -n <namespace of the CNF> results in the following error

admin [ ~ ]$ kubectl get pods -n <CNF namespace>

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:tca-rbac:dbada653-281a-4aa5-8215-85a1911abc97" cannot list resource "pods" in API group "" in the namespace "<CNF namespace>"

Environment

VMware Telco Cloud Automation 2.0
VMware Telco Cloud Automation 2.3
VMware Telco Cloud Automation 2.1
VMware Telco Cloud Automation 2.2
VMware Telco Cloud Automation 2.1.1
VMware Telco Cloud Automation 2.0.1

Cause

This is caused during the CNF terminate procedure where the required roles and role-bindings are deleted along with the CNF namespace.  During the re-instantiation they are not re-created.

Resolution

This problem will be fixed in TCA 3.0. However, the problem can be resolved manually, even on TCA 2.3, by following the steps listed below in the Workaround section of this article.

Workaround:
Complete the following steps:

1. Go to the Virtual Infrastructure page 
2. Select the VIM where the problematic CNF is deployed
3. Open a terminal to that VIM
4. Apply the following command to replace CNF identifier with the correct CNF identifier: kubectl delete serviceaccount -n tca-rbac -l 'tca.vmware.com/cnfId=CNF identifier'
5. Clear the proxy's cached service account in one of two ways:
   1. Wait at least 10 minutes without opening terminal / kubectl on the problematic CNF, or
   2. Restart proxy service on TCA-CP, this will clear the service account cache immediately.  Do this by connecting an SSH session to the TCA-CP where the VIM is located and running the following command:

 sudo systemctl restart proxy

NOTE: The CNF Identifier can be found by expanding the arrow of the CNF within the Network Function Inventory page. 

Additional Information

Do not delete the CNF namespace and resources in the CNF namespace which are created by TCA. Either remove this step from CNF post-terminate workflow or remove it from the manual re-instantion procedure.