"error: You must be logged into the server (Unauthorized)" returned from workload cluster when using Pinniped and Dex authentication

Products

• Tanzu Kubernetes Grid

Issue/Introduction

• TKGM workload clusters are configured with Pinniped authenticating to Dex for LDAP connectivity
• Users attempting to run commands against workload cluster using kubectl commands (with downloaded, valid kubeconfig) see errors like:
  
  error: You must be logged in to the server (Unauthorized)
  
  AND:
  
  Error: pinniped-auth login failed: exit status 1
Error: exit status 1
Error: could not complete concierge credential exchange: could not login: the server could not find the requested resource
  
  
• From SSH to the workload cluster Control Plane VM's, users are able to successfully run kubectl commands. Kube-api server pods are running.
• All Pinniped and Dex pods are running on workload cluster
• kube-api server pod logging from the workload cluster shows intermittent errors like:
  
  "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"
  
  
• When running the timedatectl command from SSH to workload cluster nodes directly, the Local time is more than 2 minutes different than the RTC time, for example:
  
  

  # timedatectl
  
  

  Local time: Wed 2024-05-15 10:30:00 UTC
Universal time: Wed 2024-05-15 10:30:00 UTC
RTC time: Wed 2024-05-15 10:36:00 UTC
Time zone: UTC (UTC, +0000)

  System clock synchronized: no

  NTP service: active

  RTC in local TZ: no

Cause

The authentication failures occur because the server from which users attempt to login is greater than 2 minutes in time ahead of the workload cluster node's local time. The token issued by Dex defaults to a 2 minute expiry, leading to a failure in authentication. If the source and destination server's times are greater than 2 minutes apart, the token can never be validated.

Resolution

1. Configure an NTP server via ytt customization in plan-based TKGM clusters according to Legacy Cluster Configuration with ytt documentation
2. Configure an NTP server via NTP_SERVER variable in class-based TKGM clusters according to Configuration File Variable Reference documentation
3. For remediation of existing cluster nodes, Enable Periodic Time Synchronization in vSphere on all Control Planes and Worker nodes in the impacted workload cluster.