NSX Application Platform (NAPP) Error - Unable to connect, system has encountered a connectivity issue due to the expiry of Kubernetes Configuration. Update the Kubernetes Configuration to resolve.

Products

VMware NSX

Issue/Introduction

This article provides steps to replace the default Kubernetes configuration file with a non-expiring token after the NSX Application Platform deployment. The steps in this article apply to the NSX Application Platform (NAPP) deployments that were originally installed with a default kubeconfig file obtained with the command:

cat /root/.kube/config > default_kubeconfig

Symptoms:

From NSX-T 3.2 with NSX Application platform (NAPP) deployed.
This issue will arise if NAPP is deployed on top of vSphere Supervisor.

The following error banner is displayed when accessing NAPP:

Unable to connect, system has encountered a connectivity issue due to the expiry of Kubernetes Configuration. Update the Kubernetes Configuration to resolve.

While connected to a NSX manager, napp-k commands can be used to run kubectl commands against the NAPP cluster. However in this scenario, because the napp account's token has expired, napp-k commands will fail with an error message similar to the below:
- ```
 You must be logged in to the server. (Unauthorized)
```

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

The default kubeconfig file in a vSphere Supervisor Workload Cluster contains a token which expires after ten hours by default, aforementioned warning for the NSX Application Platform in the NSX UI.

When the token is expired for the NAPP account, napp-k commands (alias equivalent to kubectl) on NSX managers will fail.

Resolution

To resolve this issue, the vSphere Supervisor Workload cluster token for the NAPP account needs to be recreated for renewal and associated with a serviceaccount so that it does not expire.

The below steps assume access to the Supervisor cluster as root and access to one of the NSX managers to run napp-k config view commands.

It is also assumed that the NSX manager has the previous kubeconfig with the expired token for the desired account NAPP uses to communicate with vSphere Supervisor.

These steps will advise on creating a serviceaccount with cluster-admin role for the NAPP account, generating the token and replacing the expired token in the NSX manager's kubeconfig with the renewed token.

SSH into one of the NSX managers as an user that can run napp-k commands.

Perform a config view with napp-k to locate the name of the NAPP user configured in the environment:

napp-k config view -ojsonpath='{.users}'

[{"name":"wcp:<cluster VIP address>:<nappuser>@vsphere.local"

Connect into the NAPP cluster context:
- See Connect to a TKG Service Cluster as a vCenter SSO User with Kubectl
  - If you are unable to connect into the NAPP cluster context, please reach out to VMware by Broadcom Technical Support referencing this KB article for assistance.
Create the service account for the NAPP user (from Step 2) in the kube-system namespace:
- ```
kubectl create serviceaccount <nappuser> -n kube-system
```

Create the service account's clusterrolebinding with the role of cluster-admin to be associated with the NAPP user:

kubectl create clusterrolebinding <nappuser> --serviceaccount=kube-system:<nappuser> --clusterrole=cluster-admin

Create the YAML file used for generating the new token for the NAPP user, replacing <nappuser> with the user from Step 2:

```
vi <nappuser>token.yaml
```

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
   name: <nappuser>
   namespace: kube-system
   annotations:
      kubernetes.io/service-account.name: "<nappuser>"

Generate the new token by using kubectl apply on the token yaml created in the previous step:
- Replace <nappuser>token.yaml with the name of the file created in the previous step:
- ```
kubectl apply -f <nappuser>token.yaml
```
Retrieve the decrypted new token generated from the previous steps, replacing <nappuser> with the user from Step 2:
- ```
kubectl get secret <nappuser> -n kube-system -ojsonpath='{.data.token}' | base64 -d
```
Return to one of the NSX managers as an user that can run napp-k commands
Locate the the kubeconfig file used by NAPP:
- ```
alias napp-k

alias napp-k 'kubectl --config=<kubeconfig directory>/.kube/config -n nsxi-platform'
```
- For example, this may be: /config/vmware/napps/.kube/config
Take a backup of this kubeconfig file, replacing <nappuser> with the user from Step 2:
- ```
cp <kubeconfig directory>/.kube/config <nappuser>-kubeconfig-backup
```
Edit the existing kubeconfig file and replace all token entries with the decrypted token value from Step 10:
- Tip: In vi, you can move the cursor over to the start of the existing, expired token and use d$ to erase only the current line.
- ```
vi <kubeconfig directory>/.kube/config

users:
- name: wcp:<cluster VIP>:<nappuser>@vsphere.local
  user:
    token: <token>
- name: wcp:<cluster VIP>:<nappuser>@vsphere.local
  user:
    token: <token>
```
Confirm that napp-k commands work:
- ```
napp-k get pods
```
If napp-k commands work, the new token was created and set in the kubeconfig successfully. Proceed with the next step.
- Otherwise, check that the previous steps were performed properly for the correct NAPP user.
Copy the entire contents of the kubeconfig file that has been edited with the new token in a new text file and save it:
- ```
cat <kubeconfig directory/.kube/config
```
Upload the newly generated kubeconfig file with a non-expiring token to the NSX Application Platform.

a. On the NSX UI Home page, navigate to System > NSX Application Platform.

b. From the Actions drop-down menu, select Settings.

c. In the Configuration section, click the EDIT link.

d. In the Upload File field, click SELECT, select the newly generated config file with a non-expiry token, click Open, and click Upload.

e. Ensure that the file is successfully uploaded and click NEXT.

f. Click the RUN PRECHECKS button, to ensure the environment still satisfies all the NSX Application Platform deployment requirements.

g. Click NEXT to continue.

h. Click CLOSE.

Additional Information

Impact/Risks:
This warning does not have an impact on the functionality of the NSX Application Platform nor any of the security features currently installed.

However, you will need to generate a valid (not expired token) every time you want to:

Deploy the NSX Application Platform
Upgrade the NSX Application Platform
Delete the NSX Application Platform