TKGi-MC doesn't show Cluster Overview with token errors

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

TKGi Management Console fails to load the Cluster Overview with errors:

'Failed to retrieve current K8s Cluster summary. cannot get cluster details: cannot get cluster namespaces: Get "https://<NSX-LB-FIP>:8443/api/v1/namespaces": failed to refresh token: oauth2: "invalid_token" "Could not verify token signature."'

'Failed to retrieve current K8s Cluster Volumes. cannot get namespaces of cluster <cluster-id>: Get "https://<NSX-LB-FIP>:8443/api/v1/namespaces": failed to refresh token: oauth2: "invalid_token" "Could not verify token signature."'

'Failed to retrieve current K8s Cluster summary. cannot get cluster details: cannot get cluster namespaces: Get "https://<NSX-LB-FIP>:8443/api/v1/namespaces": could not persist new tokens: open : no such file or directory"'

'Failed to retrieve current K8s Cluster Volumes. cannot get namespaces of cluster <cluster-id>: Get "https://<NSX-LB-FIP>:8443/api/v1/namespaces": could not persist new tokens: open : no such file or directory"'

Cause

This can be caused by errors inside the TKGi-MC pks-mgmt-server and/or the TKGi API VM's UAA.

Resolution

1. Login to TKGi-MC VM as root user.
   
   
2. Restart the pks-mgmt-server:
   # systemctl restart pks-mgmt-server
   
   
3. Login to OpsMan VM or to any jumpbox with Bosh CLI client.
   
   
4. Restart uaa job in TKGi API VM:
   # bosh -d pivotal-container-service-<id> ssh pivotal-container-service -c "sudo monit restart uaa"
   
   where "pivotal-container-service-<id>" is de TKGi API's Bosh deployment name.
   
   

If the above doesn't resolve the issue, please open a Support Request with Tanzu Support and provide the following logs:

• TKGi API VM's logs:
  # bosh -d pivotal-container-service-<id> logs pivotal-container-service
  
  
• From TKGi-MC VM, collect docker logs for all its components:
  # docker logs pks-mgmt-server &> pks-mgmt-server.log
# docker logs pks-loginsight &> pks-loginsight.log
# docker logs pks-frontend &> pks-frontend.log
# docker logs pks-ui-server &> pks-ui-server.log
  
  
• HAR file from the browser where the TKGi-MC UI error is seen.
  • From the browser, go to the page where you're getting the error. Then open the Developer Tools (Ctrl+Shift+I in Chrome).
  • Go to Network. Now reload the page. This will show all the requests sent from the browser.
  • Click on "Export HAR" and upload the file to the SR case.