vmop-controller-manager reports "401 Unauthorized" error

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:

• vSphere Client shows the following K8s events errors:
  
  Name - Kind - Type - Message - Reason - Component <>-control-plane-<> - VirtualMachine - Warning - GET https://<>:443/rest/vcenter/cluster/modules/vm/<>/members: 401 Unauthorized - UpdateFailure - vmware-system-vmop/vmware-system-vmop-controller-manager-<>/virtualmachine-controller

 

• vmop-controller-manager logs show "401 Unauthorized" errors on Supervisor Control plane node:
  log location: /var/log/pods/vmware-system-vmop_vmware-system-vmop-controller-manager-5b8c97598d-nt5mv_8b7565c7-8d3f-4133-87b5-c1a4531ee1fe/manager/xx.log

YYYY-MM-DDTHH:MM:SS.#######Z stderr F E1018 ##:##:21.688666    1 contentsource_controller.go:297] controllers/ContentSource "msg"="error listing images from provider" "error"="get library items failed for: GET https://<>:443/rest/com/vmware/content/library/item?library_id=<>: 401 Unauthorized" "clProviderName"="<>" "clProviderUUID"="<>"

YYYY-MM-DDTHH:MM:SS.#######Z stderr F E1018 ##:##:21.688692    1 contentsource_controller.go:328] controllers/ContentSource "msg"="Error listing VirtualMachineImages from the content provider" "error"="get library items failed for: GET https://<>:443/rest/com/vmware/content/library/item?library_id=<>: 401 Unauthorized" "contentSourceName"="<>"

YYYY-MM-DDTHH:MM:SS.#######Z stderr F E1018 ##:##:21.688722    1 contentsource_controller.go:343] controllers/ContentSource "msg"="failed to difference images" "error"="get library items failed for: GET https://<>:443/rest/com/vmware/content/library/item?library_id=<>: 401 Unauthorized"

 

Environment

VMware vSphere 7.0 with Tanzu

Cause

The "401 Unauthorized" errors in vmop-controller-manager suggest an authentication failure with the Supervisor cluster.
This authentication is necessary for vmop-controller-manager to monitor the TKC clusters and ensure they align with their desired state.

Resolution

Validate if the credentials are in sync,

• Check the credentials stored in wcp-vmop-sa-vc-auth secret:

SSH to Supervisor control Plan node from vCenter.

root@vcenter [ ~ ]# /usr/lib/vmware-wcp/decryptK8Pwd.py

Cluster: domain-c8:###################
IP: <Ip address>
PWD: ###########

Execute the following command on the Supervisor Control Plane node to obtain the username and password:

kubectl get secret -n vmware-system-vmop wcp-vmop-sa-vc-auth -oyaml -ojsonpath='{.data.username}' |base64 -d; echo

kubectl get secret -n vmware-system-vmop wcp-vmop-sa-vc-auth -oyaml -ojsonpath='{.data.password}' |base64 -d; echo

Example:

root@######################## [ ~ ]# kubectl get secret -n vmware-system-vmop wcp-vmop-sa-vc-auth -oyaml -ojsonpath='{.data.username}' |base64 -d; echo

wcp-vmop-user-domain-c8-########-####-####-####-##########@vsphere.local

root@######################## [ ~ ]# kubectl get secret -n vmware-system-vmop wcp-vmop-sa-vc-auth -oyaml -ojsonpath='{.data.password}' |base64 -d; echo

################

• Use the credentials obtained from the previous command to authenticate and access the vSphere Client.
  • If an "Invalid credentials" error occurs, restart the WCP service from vCenter.
    

     

    service-control --restart wcp

• Wait for a few hours until the next password renewal by WCP occurs. After it's done, you'll see that wcp-vmop-sa-vc-auth password has changed:
  Execute the following command on the Supervisor Control Plane node to confirm password change:  
  

root@################### [ ~ ]# kubectl get secret -n vmware-system-vmop wcp-vmop-sa-vc-auth -oyaml -ojsonpath='{.data.password}' |base64 -d; echo

• Try again to login to vSphere Client with the new password obtained from above cmd.
• If the issue persists, it's recommended to follow the below iKB for further troubleshooting: 
  TKC Cluster New Worker node deployment fails with error - failed to find image 401 unauthorized