Install Cluster Essentials on Kubernetes cluster for TAP installation failed to pull and unpack image with 401 Unauthorized

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Customer is trying to install Cluster Essentials on Kubernetes cluster following the doc Deploying Cluster Essentials. However, they are failed at the step run install.sh with below error message:

kapp: Error: waiting on reconcile deployment/kapp-controller (apps/v1) namespace: kapp-controller:
  Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "kapp-controller-6645c4d7bc" has timed out progressing.))

To further troubleshooting on this, we need to get the kapp-controller pod status and check its events. 
$ kubectl get pod -n kapp-controller

NAME                               READY   STATUS             RESTARTS   AGE
kapp-controller-6645c4d7bc-wwght   0/2     ImagePullBackOff   0          34m

$ kubectl describe pod -n kapp-controller kapp-controller-6645c4d7bc-wwght

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  35m                 default-scheduler  Successfully assigned kapp-controller/kapp-controller-6645c4d7bc-wwght to aks-devtoznrun1-23946051-vmss000001
  Normal   Pulling    34m (x3 over 35m)   kubelet            Pulling image "registry.tanzu.vmware.com/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:a9e4a584f03c5aaceae405d8a741b9a3681c23398023e589a6514501f1ac1bc2"
  Warning  Failed     34m (x3 over 35m)   kubelet            Failed to pull image "registry.tanzu.vmware.com/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:a9e4a584f03c5aaceae405d8a741b9a3681c23398023e589a6514501f1ac1bc2": rpc error: code = Unknown desc = failed to pull and unpack image "registry.tanzu.vmware.com/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:a9e4a584f03c5aaceae405d8a741b9a3681c23398023e589a6514501f1ac1bc2": failed to resolve reference "registry.tanzu.vmware.com/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:a9e4a584f03c5aaceae405d8a741b9a3681c23398023e589a6514501f1ac1bc2": pulling from host registry.tanzu.vmware.com failed with status code [manifests sha256:a9e4a584f03c5aaceae405d8a741b9a3681c23398023e589a6514501f1ac1bc2]: 401 Unauthorized
  Warning  Failed     34m (x3 over 35m)   kubelet            Error: ErrImagePull
  Normal   BackOff    34m (x5 over 35m)   kubelet            Back-off pulling image "registry.tanzu.vmware.com/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:a9e4a584f03c5aaceae405d8a741b9a3681c23398023e589a6514501f1ac1bc2"
  Warning  Failed     34m (x5 over 35m)   kubelet            Error: ImagePullBackOff
  Warning  Failed     34m (x3 over 35m)   kubelet            Error: ImagePullBackOff
  Normal   BackOff    7s (x152 over 35m)  kubelet            Back-off pulling image "registry.tanzu.vmware.com/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:a9e4a584f03c5aaceae405d8a741b9a3681c23398023e589a6514501f1ac1bc2"

Environment

Product Version: 1.5

Resolution

From the pod Event output, we can see the issue here is that image pulling from host registry.tanzu.vmware.com failed with status code "401 Unauthorized". Customer can use below command to get the secret with decode credentials to verify if the credential string is correct or not. 

kubectl get secret kapp-controller-registry-creds -n kapp-controller -ojson | jq -r '.data.".dockerconfigjson"' | base64 -d

If the credential is wrong, you need to export below parameters to correct the credential.

export INSTALL_REGISTRY_USERNAME=TANZU-NET-USER
export INSTALL_REGISTRY_PASSWORD=TANZU-NET-PASSWORD

Note: Suggest to enclose the TANZU-NET-PASSWORD in "" (Double quotes).