Tanzu Hub Registry VM shows unauthorized errors when running kubectl commands after failed upgrade

Products

• VMware Tanzu Platform - Hub

Issue/Introduction

• While upgrading the Tanzu Hub tile, the upgrade hangs for fails.
• While attempting investigation of the failure, kubectl commands run from the registry VM fail with "unauthorized" errors, or, errors like:
  

  
error: You must be logged in to the server (Unauthorized)

E0121 15:56:22.427329 1568345 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"

Environment

This was seen on Tanzu Hub environments during upgrade from 10.2.1 to 10.3.0.

Cause

During the upgrade from Tanzu hub 10.2.1 to 10.3, the password for the admin use in kubeconfig on the Registry VM is updated. Depending on the stage of upgrade at which the failure occurs, the password might be updated in Credhub, but not propagated to the kubeconfig on the Registry VM yet.

Resolution

Manually gather the password from Credhub and update the /var/vcap/jobs/hubsm-install/config/kubeconfig file:

 

1. Use the Accessing Bosh Credhub with Credhub CLI documentation for reference to run Credhub CLI commands against the Hub deployment.
2. Gather the admin password:
   
   

   credhub get -n /p-bosh/hub-<DEPLOYMENT_ID>/kube-apiserver-admin-password
   
   

3. Backup the /var/vcap/jobs/hubsm-install/config/kubeconfig file:
   
   sudo cp /var/vcap/jobs/hubsm-install/config/kubeconfig /var/vcap/jobs/hubsm-install/config/kubeconfig.bak


4. Edit the /var/vcap/jobs/hubsm-install/config/kubeconfig file, change the token value to match the password gathered from Credhub CLI