Kubectl commands failed with the error "failed to verify certificate: x509: certificate signed by unknown authority" in TKGI

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

We will notice below error while running kubectl commands in TKGI

E1123 05:33:21.223461       7 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, oidc: verify token: failed to verify signat  ure: fetching keys oidc: get keys failed Get \"https://api.pks.example.com:8443/token_keys\": tls: failed to verify certificate: x509: certificate signed  by unknown authority]"

Unable to connect to the server: Get "https://api.pks.example.com:8443/oauth/token/.well-known/openid-configuration": x509: certificate signed by unknown authority
 

This happens when OIDC is enabled in the TKGI environment and TKGI VM is upgraded but cluster is not yet upgraded.

Find the OIDC cert using the below bosh command.

bosh -d service-instsance-xxxxxx manifest 

oidc:
 >         ca: |

In addition we can ssh to the master nodes in the below location and find the cert.

/var/vcap/jobs/kube-apiserver/config/oidc-ca.pem

Environment

TKGI with OIDC

Cause

We need to do a upgrade-cluster after rotating pks-api certificate for OIDC cases, since the pks-api/UAA certificate has changed.

Resolution

Upgrade the TKGI clusters.