TKC Cluster New Worker node deployment fails with error - failed to find image 401 unauthorized

Products

• VMware vSphere with Tanzu

Issue/Introduction

• Deploying new TKC worker node fails. While describing the failed node you will see the below message
  
  Events:
  Type     Reason                 Age                     From                                                                                                 Message
  ----     ------                 ----                    ----                                                                                                 -------
  Warning  CreateOrUpdateFailure  2m44s (x14 over 3m26s)  vmware-system-vmop/vmware-system-vmop-controller-manager-xxxxxxxxxx-xxxxx/virtualmachine-controller  failed to find image: <UUID>: GET https://<VC-FQDN>:443/rest/com/vmware/content/library/item/id:<UUID>: 401 Unauthorized

 
• VMOP logs from Supervisor Cluster will show below errors - /var/log/pods/vmware-system-vmop_vmware-system-vmop-controller-manager-5b8c97598d-nt5mv_8b7565c7-8d3f-4133-87b5-c1a4531ee1fe/manager/xx.log 

stderr F E0731 <Time-stamp> 1 vmprovider_vm.go:330] "CreateVirtualMachine failed" err="failed to find image: <UUID>: GET https://<VC-FQDN>:443/rest/com/vmware/content/library/item/id:<UUID>: 401 Unauthorized" logger="vsphere" vmName="<guest-cluster-namespace-name>/<newly_deployed failed_worker_node_name>"
  
  
  
• ENVOY logs from vCenter Server will show below errors - /var/log/vmware/envoy/envoy-x.log

info envoy[130422] [Originator@6876 sub=Default] <Date_TIME> GET /rest/com/vmware/content/library/item/id:<UUID> 401 via_upstream - 0 316 gzip 3 2 0 <Supervisor-Management-IP-address>:54882 HTTP/2 TLSv1.2 <VC-IP-address>:443 127.0.0.1:37614 HTTP/1.1 - 127.0.0.1:12346 - -

Cause

This issue can happen if the vpxd-<machine-id> is missing in ActAsUsers group. 

Resolution

• Take SSH/Putty session to the vCenter Server run below command to find the vCenter Server machine-id
  
  /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
  
  
  
• Add the missing vpxd user to ActAsUsers group by running below command
  
  /usr/lib/vmware-vmafd/bin/dir-cli group modify --name 'ActAsUsers' --add "vpxd-<machine-id from previous output>" --login 'SSO_ADMIN_USER'