tanzu isolated-cluster upload-bundle command fails with unexpected status code 401 Unauthorized
search cancel

tanzu isolated-cluster upload-bundle command fails with unexpected status code 401 Unauthorized

book

Article ID: 380756

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Management

Issue/Introduction

  • tanzu isolated-cluster upload-bundle command is failing with unexpected status code 401 Unauthorized every time we ran it on different image every time.

tanzu isolated-cluster upload-bundle --source-directory <SOURCE-DIRECTORY> --destination-repo <DESTINATION-REGISTRY> --ca-certificate <SECURITY-CERTIFICATE>

# tanzu isolated-cluster upload-bundle --source-directory /root/tkg/ --destination-repo harbor-ip/tkg --insecure -v 9 --debug


Error: error while uploading the images: HEAD https://harbor.ip:443/v2/tkg/kube-scheduler/blobs/sha256:3996ddec0b8122956c7d44b0ebf3cbfec8720a97fd6470614b5c1054c24e7d6f: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

  • When validating image tags uploaded to harbor you find that there is one or more images missing.

Use imgpkg cli to list all the tags for the images and compare the output to the images in the  publish-images-from tar.yaml file.

Ex: v1.28.7_vmware.1 tag is missing on the harbor for the kube-scheduler image

# imgpkg tag list -i harbor-ips:/tkg/kube-scheduler
Tags

Name
v1.26.14_vmware.1
v1.27.11_vmware.1

2 tags

# cat publish-images-fromtar.yaml | grep kube-scheduler

kube-scheduler-v1.26.14_vmware.1.tar: kube-scheduler
kube-scheduler-v1.27.11_vmware.1.tar: kube-scheduler
kube-scheduler-v1.28.7_vmware.1.tar: kube-scheduler

Note: The publish-images-from tar.yaml file will be located in the same directory where the image bundle in the form of TAR files are located. The YAML file defines the mapping between the images and the TAR files.

  • Running the tanzu isolated-cluster upload-bundle command again will upload the failed image tag but still will fail with error unexpected status code 401 Unauthorized, but this time all the image tags got uploaded to the harbor.

Error: error while uploading the images: HEAD https://harbor.ip:443/v2/tkg/tkr-bom/blobs/sha256:d05085c78fcfd3438ec9aa0d4ac772d1e82fbe63795087b21300c7278bde0c39: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

imgpkg tag list -i harbor.pks-06.slot-38.tanzu-nsx-labs.vmware.com:/tkg/tkr-bom
Tags

Name
v1.26.14_vmware.1-tiny.1-tkg.1
v1.26.14_vmware.1-tkg.2
v1.27.11_vmware.1-tiny.1-tkg.1
v1.27.11_vmware.1-tkg.3
v1.28.7_vmware.1-tiny.1-tkg.1
v1.28.7_vmware.1-tkg.3

6 tags

Succeeded
root@opsmgr-06-slot-38-tanzu-nsx-labs:~/tkg# cat publish-images-fromtar.yaml | grep tkr-bom
tkr-bom-v1.26.14_vmware.1-tiny.1-tkg.1.tar: tkr-bom
tkr-bom-v1.26.14_vmware.1-tkg.2.tar: tkr-bom
tkr-bom-v1.27.11_vmware.1-tiny.1-tkg.1.tar: tkr-bom
tkr-bom-v1.27.11_vmware.1-tkg.3.tar: tkr-bom
tkr-bom-v1.28.7_vmware.1-tiny.1-tkg.1.tar: tkr-bom
tkr-bom-v1.28.7_vmware.1-tkg.3.tar: tkr-bom

 

  • Multiple attempts to run the tanzu isolated-cluster upload-bundle command will fail with the same unexpected status code 401 Unauthorized, even when all the images are already uploaded to the registry.
  • The project 'Quota Size' won't increase and the project 'Repositories Count' will remain the same regardless of how many times the tanzu isolated-cluster upload-bundle command.

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Tanzu Kubernetes Grid Management (TKGm).

Cause

  • This is a known issue affecting uploading images using tanzu isolated-cluster upload-bundle command due to harbor connection limit.

Resolution

  • Although the tanzu isolated-cluster upload-bundle command will continue to fail with error "unexpected status code 401 Unauthorized", you will find after validating, that all the images are uploaded to the Harbor.  The project 'Quota Size' not increasing and the project 'Repositories Count' remaining the same with every upload attempt shows that the images are uploaded.
  • Creating the MGMT cluster should complete successfully since all the images are uploaded to the harbor

Additional Note: In case where there are images missing after following the validation steps, you can use the following command to force upload the image to the harbor.

Ex:

imgpkg copy --tar ${source_dir}/tkr-vsphere-nonparavirt-v1.25.7_vmware.2-tkg.1.tar --to-repo <remote-harbor-url>/tkr-vsphere-nonparavirt --registry-ca-cert-path ${harbor-cert-path}/ca.crt

 

Powered by Wolken Software