Tanzu Kubernetes Grid 1.x Authentication error using LDAP "You must be logged in to the server (Unauthorized)" is displayed on the commandline"

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:
Intermittent unavailability of the authentication with error "You must be logged in to the server (Unauthorized)" is displayed on the commandline". Users authenticate via pinniped/dex against our LDAP server and login to the workload cluster
 

 

Environment

VMware Tanzu Kubernetes Grid 1.x

Cause

This error message can appear for a number of reasons but one possible cause is a significant time skew between workload cluster nodes and all management cluster nodes.

This "unexpected validation error" is encountered in one specific case: the API server believes that the token is not valid yet.  This will only occur when nodes have a significant clock skew that causes tokens issued from one node to be considered not valid yet by another node.

Many instances showing below error:

1398:E1026 13:40:56.079238 1 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, Token could not be validated.]" and 52:E1026 12:48:11.627682 1 claims.go:126] unexpected validation error: *errors.errorString

 

Resolution

Fixing time skew between workload cluster nodes and all management cluster nodes.

See NTP in Preparing to Deploy Management Clusters or  Configuring NTP without DHCP Option 42

Additional Information

These error messages originate from  https://github.com/kubernetes/kubernetes/blob/10988997f225447f89841bac08e8848852d7cb55/pkg/serviceaccount/claims.go#L126-L127