TKC Cluster New Worker node deployment fails with error - failed to find image 401 unauthorized
search cancel

TKC Cluster New Worker node deployment fails with error - failed to find image 401 unauthorized

book

Article ID: 374782

calendar_today

Updated On:

Products

VMware vSphere with Tanzu

Issue/Introduction

  • Deploying new TKC worker node fails. While describing the failed node you will see the below message

    Events:
      Type     Reason                 Age                     From                                                                                                 Message
      ----     ------                 ----                    ----                                                                                                 -------
      Warning  CreateOrUpdateFailure  2m44s (x14 over 3m26s)  vmware-system-vmop/vmware-system-vmop-controller-manager-xxxxxxxxxx-xxxxx/virtualmachine-controller  failed to find image: <UUID>: GET https://<VC-FQDN>:443/rest/com/vmware/content/library/item/id:<UUID>: 401 Unauthorized

     
  • VMOP logs from Supervisor Cluster will show below errors - /var/log/pods/vmware-system-vmop_vmware-system-vmop-controller-manager-5b8c97598d-nt5mv_8b7565c7-8d3f-4133-87b5-c1a4531ee1fe/manager/xx.log 

    stderr F E0731 <Time-stamp> 1 vmprovider_vm.go:330] "CreateVirtualMachine failed" err="failed to find image: <UUID>: GET https://<VC-FQDN>:443/rest/com/vmware/content/library/item/id:<UUID>: 401 Unauthorized" logger="vsphere" vmName="<guest-cluster-namespace-name>/<newly_deployed failed_worker_node_name>"



  • ENVOY logs from vCenter Server will show below errors - /var/log/vmware/envoy/envoy-x.log

    info envoy[130422] [Originator@6876 sub=Default] <Date_TIME> GET /rest/com/vmware/content/library/item/id:<UUID> 401 via_upstream - 0 316 gzip 3 2 0 <Supervisor-Management-IP-address>:54882 HTTP/2 TLSv1.2 <VC-IP-address>:443 127.0.0.1:37614 HTTP/1.1 - 127.0.0.1:12346 - -

Cause

This issue can happen if the vpxd-<machine-id> is missing in ActAsUsers group. 

Resolution

  • Take SSH/Putty session to the vCenter Server run below command to find the vCenter Server machine-id

    /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost


  • Add the missing vpxd user to ActAsUsers group by running below command

    /usr/lib/vmware-vmafd/bin/dir-cli group modify --name 'ActAsUsers' --add "vpxd-<machine-id from previous output>" --login 'SSO_ADMIN_USER'