Cannot access the clusters using tkg/tanzu cli commands in Tanzu Kubernetes Grid
search cancel

Cannot access the clusters using tkg/tanzu cli commands in Tanzu Kubernetes Grid

book

Article ID: 316961

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:
  • Your Tanzu Kubernetes Grid (TKG) installation was created over one year ago.
  • Your TKG installation has been upgraded at least once since the initial installation.
  • You see messages similar to the following when running tkg or tanzu CLI commands:

tkg get clusters --include-management-cluster -v 9

Default BOM file is: bom-1.2.1+vmware.1.yaml
unable to find image capdControllerImage in BOM file
Using configuration file: /home/ubuntu/.tkg/config.yaml
Checking cluster reachability...
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retrying
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retrying
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retrying
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retrying
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retryingError: : unable to get cluster client while listing tkg clusters: unable to get client: Failed to invoke API on cluster : the server has asked for the client to provide credentialsDetailed log about the failure can be found at: /tmp/tkg-20210706T113804512680725.log

 

  • You can ssh to a control plane node and use the /etc/kubernetes/admin.conf file as your kubeconfig file to successfully run kubectl commands. 
  • You can use the sudo kubeadm alpha certs check-expiration command to validate the certificate expiry of all the system components from the control plane node. Note: You will see output similar to the following:

[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Feb 25, 2022 16:03 UTC   234d                                    no
apiserver                  Feb 25, 2022 16:03 UTC   234d            ca                      no
apiserver-etcd-client      Feb 25, 2022 16:03 UTC   234d            etcd-ca                 no
apiserver-kubelet-client   Feb 25, 2022 16:03 UTC   234d            ca                      no
controller-manager.conf    Feb 25, 2022 16:03 UTC   234d                                    no
etcd-healthcheck-client    Feb 25, 2022 16:03 UTC   234d            etcd-ca                 no
etcd-peer                  Feb 25, 2022 16:03 UTC   234d            etcd-ca                 no
etcd-server                Feb 25, 2022 16:03 UTC   234d            etcd-ca                 no
front-proxy-client         Feb 25, 2022 16:03 UTC   234d            front-proxy-ca          no
scheduler.conf             Feb 25, 2022 16:03 UTC   234d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Feb 23, 2031 16:03 UTC   9y              no
etcd-ca                 Feb 23, 2031 16:03 UTC   9y              no
front-proxy-ca          Feb 23, 2031 16:03 UTC   9y              no

  • When you examine the relevant certificate data in the ~/.kube-tkg/config file, you see that the noted certificate is expired. Note: The certificate data in the client-certificate-data section is base64-encoded but can be examined by issuing a command similar to the following:

    echo LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJWDl3cDRuZ1JPUkF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TVRBeU1qVXhOVFU0TURWYUZ3MHlNakF5TWpVeE5qQXpNRFphTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXd3clZCRnJ4Q3p1WXNhTzIKOWVEU0N3MTdqR2lleE80RG9vUjFQR3JNSWVDZWN6RVByVWJRTDFjUDV6ZFRWUk9yc3hXUzdRL2ZzamJlbithTApmSjh1RDU3YkNrYUR3ZGFIR3owbnJhbEMyejhsWXEvQ2VzYmZrUlBJanl0NXUvTFFSdTc4ZFVxK0RRZnY0YVNPCkNhbDUrenk4eGlSM25EY1JEbHJPT1BvUDJKbHBZQy96dHBwakszeGQySlppdXBIa0U4dlY5WHF4K21jMFZSbmsKeHBhZTYvc1BSL2N5STVkZDlmSkYydVduMzRZdzBwNVBIUEwxSmRlSy9WMjJ0Y1pPTFg5MFVQbm1JRWI3eXpGagpXbVpyT2F1WkcwTTVxZ2R5a0pPOGV3ekZzVHpDakpuZzJneXliQ0lOSUI4bVMwY2NnenRseEcyWHJDamxxUlc5Cm1kM2Erd0lEQVFBQm95Y3dKVEFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEUTZwODFhb0RDN0paVThKTFJwUno4Skk3eDhDbldFRDBaTwo1ekUrdTlQVm5tM010M0V4dUpjeFkvQ2JXR0dLQkpMQWlKR0FRMkFsbU9aVS9scmNEd09rL3FKSGpNQ0UyajY2ClMwWTI3RVh6eVVzbjFubUlEUzhIVVR6bEFPc0NXZkVTZHZkK3VjSzB6aFFxVGNQRWhiSUlBaTRhWDBmdFlpcSsKTjZnaFFTNTR1NHJJUW1RVUpxcTJ0cUdtY2V3d0U4TDJSbmdGaEJ2cUpXdXFKajBCbHcwZi9sMENEU3hKTlMzUwpLRURtU2h3UVo3UmN5dDVQdUZ0NW94cjBwb0lZSUsvUkxuakVOVFFWYnhoeHplaXJ0bDBpSXlCck5QZVlzMHEvClhiK3c2WWpIQzRNdWhzZTlMdnZMYVRJdHphd1RRZTRDT0ZRWW5GTkpDL054WFhiNlhuVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= | base64 -d | openssl x509 -in - -text -noout

    Note: You will see output similar to the following:

    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 6907441981133240592 (0x5fdc29e278113910)
            Signature Algorithm: sha256WithRSAEncryption
            Issuer: CN = kubernetes
            Validity
                Not Before: Jul 5 15:58:05 2020 GMT
                Not After :  Jul 4 15:58:05 2021 GMT
            Subject: O = system:masters, CN = kubernetes-admin



Environment

VMware Tanzu Kubernetes Grid Plus 1.x

Resolution

This is a known issue affecting TKG 1.2 and 1.3. There is currently no resolution.

Workaround:
  1. From the management cluster context, issue a command similar to the following to get the correct cluster configuration data to populate the ~/.kube-tkg/config file :

kubectl -n tkg-system get secrets <cluster name>-kubeconfig -o 'go-template={{ index .data "value"}}' | base64 -d

Note: Replace <cluster name> with the name of the management cluster.
Note: You will see output similar to the following:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBD<redacted>
    server: https://192.168.100.90:6443
  name: tkg-mgmt
contexts:
- context:
    cluster: tkg-mgmt
    user: tkg-mgmt-admin
  name: tkg-mgmt-admin@tkg-mgmt
current-context: tkg-mgmt-admin@tkg-mgmt
kind: Config
preferences: {}
users:
- name: tkg-mgmt-admin
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZ<redacted>
    client-key-data: LS0tLS1CRUdJTiBSU<redacted>

Note: You can use a base64 command similar to the one noted in the Symptoms section to validate the certificate in the client-certificate-data field.

  1. Issue the cp ~/.kube-tkg/config ~/.kube-tkg/config_backup command to take a backup of the existing config file.
  2. Open the ~/.kube-tkg/config file using a text editor and replace the contents with the output from the command in Step 1.

Note: You should now be able to run tkg or tanzu CLI commands successfully.