Cannot access the clusters using tkg/tanzu cli commands in Tanzu Kubernetes Grid
Article ID: 316961
Updated On:
Products
Issue/Introduction
Symptoms:
- Your Tanzu Kubernetes Grid (TKG) installation was created over one year ago.
- Your TKG installation has been upgraded at least once since the initial installation.
- You see messages similar to the following when running tkg or tanzu CLI commands:
tkg get clusters --include-management-cluster -v 9
Default BOM file is: bom-1.2.1+vmware.1.yaml
unable to find image capdControllerImage in BOM file
Using configuration file: /home/ubuntu/.tkg/config.yaml
Checking cluster reachability...
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retrying
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retrying
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retrying
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retrying
Failed to invoke API on cluster : the server has asked for the client to provide credentials, retryingError: : unable to get cluster client while listing tkg clusters: unable to get client: Failed to invoke API on cluster : the server has asked for the client to provide credentialsDetailed log about the failure can be found at: /tmp/tkg-20210706T113804512680725.log
- You can ssh to a control plane node and use the /etc/kubernetes/admin.conf file as your kubeconfig file to successfully run kubectl commands.
- You can use the sudo kubeadm alpha certs check-expiration command to validate the certificate expiry of all the system components from the control plane node. Note: You will see output similar to the following:
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Feb 25, 2022 16:03 UTC 234d no
apiserver Feb 25, 2022 16:03 UTC 234d ca no
apiserver-etcd-client Feb 25, 2022 16:03 UTC 234d etcd-ca no
apiserver-kubelet-client Feb 25, 2022 16:03 UTC 234d ca no
controller-manager.conf Feb 25, 2022 16:03 UTC 234d no
etcd-healthcheck-client Feb 25, 2022 16:03 UTC 234d etcd-ca no
etcd-peer Feb 25, 2022 16:03 UTC 234d etcd-ca no
etcd-server Feb 25, 2022 16:03 UTC 234d etcd-ca no
front-proxy-client Feb 25, 2022 16:03 UTC 234d front-proxy-ca no
scheduler.conf Feb 25, 2022 16:03 UTC 234d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Feb 23, 2031 16:03 UTC 9y no
etcd-ca Feb 23, 2031 16:03 UTC 9y no
front-proxy-ca Feb 23, 2031 16:03 UTC 9y no
-
When you examine the relevant certificate data in the ~/.kube-tkg/config file, you see that the noted certificate is expired. Note: The certificate data in the client-certificate-data section is base64-encoded but can be examined by issuing a command similar to the following:
echo LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJWDl3cDRuZ1JPUkF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TVRBeU1qVXhOVFU0TURWYUZ3MHlNakF5TWpVeE5qQXpNRFphTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXd3clZCRnJ4Q3p1WXNhTzIKOWVEU0N3MTdqR2lleE80RG9vUjFQR3JNSWVDZWN6RVByVWJRTDFjUDV6ZFRWUk9yc3hXUzdRL2ZzamJlbithTApmSjh1RDU3YkNrYUR3ZGFIR3owbnJhbEMyejhsWXEvQ2VzYmZrUlBJanl0NXUvTFFSdTc4ZFVxK0RRZnY0YVNPCkNhbDUrenk4eGlSM25EY1JEbHJPT1BvUDJKbHBZQy96dHBwakszeGQySlppdXBIa0U4dlY5WHF4K21jMFZSbmsKeHBhZTYvc1BSL2N5STVkZDlmSkYydVduMzRZdzBwNVBIUEwxSmRlSy9WMjJ0Y1pPTFg5MFVQbm1JRWI3eXpGagpXbVpyT2F1WkcwTTVxZ2R5a0pPOGV3ekZzVHpDakpuZzJneXliQ0lOSUI4bVMwY2NnenRseEcyWHJDamxxUlc5Cm1kM2Erd0lEQVFBQm95Y3dKVEFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEUTZwODFhb0RDN0paVThKTFJwUno4Skk3eDhDbldFRDBaTwo1ekUrdTlQVm5tM010M0V4dUpjeFkvQ2JXR0dLQkpMQWlKR0FRMkFsbU9aVS9scmNEd09rL3FKSGpNQ0UyajY2ClMwWTI3RVh6eVVzbjFubUlEUzhIVVR6bEFPc0NXZkVTZHZkK3VjSzB6aFFxVGNQRWhiSUlBaTRhWDBmdFlpcSsKTjZnaFFTNTR1NHJJUW1RVUpxcTJ0cUdtY2V3d0U4TDJSbmdGaEJ2cUpXdXFKajBCbHcwZi9sMENEU3hKTlMzUwpLRURtU2h3UVo3UmN5dDVQdUZ0NW94cjBwb0lZSUsvUkxuakVOVFFWYnhoeHplaXJ0bDBpSXlCck5QZVlzMHEvClhiK3c2WWpIQzRNdWhzZTlMdnZMYVRJdHphd1RRZTRDT0ZRWW5GTkpDL054WFhiNlhuVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= | base64 -d | openssl x509 -in - -text -noout
Note: You will see output similar to the following:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 6907######240592 (0x5fd######3910)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: Jul 5 15:58:05 2020 GMT
Not After : Jul 4 15:58:05 2021 GMT
Subject: O = system:masters, CN = kubernetes-admin
Environment
Resolution
This is a known issue affecting TKG 1.2 and 1.3. There is currently no resolution.
Workaround:
- From the management cluster context, issue a command similar to the following to get the correct cluster configuration data to populate the ~/.kube-tkg/config file :
kubectl -n tkg-system get secrets <cluster name>-kubeconfig -o 'go-template={{ index .data "value"}}' | base64 -d
Note: Replace <cluster name> with the name of the management cluster.
Note: You will see output similar to the following:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tL#######TiBD<redacted>
server: https://192.###.###.###:6443
name: tkg-mgmt
contexts:
- context:
cluster: tkg-mgmt
user: tkg-###-###
name: tkg-mgmt-###@tkg-mgmt
current-context: tkg-mgmt-###@tkg-mgmt
kind: Config
preferences: {}
users:
- name: tkg-mgmt-admin
user:
client-certificate-data: LS0tL########BDRVJUSUZ<redacted>
client-key-data: LS0tLS#########iBSU<redacted>
Note: You can use a base64 command similar to the one noted in the Symptoms section to validate the certificate in the client-certificate-data field.
- Issue the cp ~/.kube-tkg/config ~/.kube-tkg/config_backup command to take a backup of the existing config file.
- Open the ~/.kube-tkg/config file using a text editor and replace the contents with the output from the command in Step 1.
Note: You should now be able to run tkg or tanzu CLI commands successfully.
Feedback
