"error: You must be logged into the server (Unauthorized)" returned from workload cluster when using Pinniped and Dex authentication
search cancel

"error: You must be logged into the server (Unauthorized)" returned from workload cluster when using Pinniped and Dex authentication

book

Article ID: 367789

calendar_today

Updated On:

Products

Tanzu Kubernetes Grid

Issue/Introduction

  • TKGM workload clusters are configured with Pinniped authenticating to Dex for LDAP connectivity
  • Users attempting to run commands against workload cluster using kubectl commands (with downloaded, valid kubeconfig) see errors like:

    error: You must be logged in to the server (Unauthorized)

    AND:

    Error: pinniped-auth login failed: exit status 1
    Error: exit status 1
    Error: could not complete concierge credential exchange: could not login: the server could not find the requested resource

  • From SSH to the workload cluster Control Plane VM's, users are able to successfully run kubectl commands. Kube-api server pods are running.
  • All Pinniped and Dex pods are running on workload cluster
  • kube-api server pod logging from the workload cluster shows intermittent errors like:

    "Unable to authenticate the request" err="[invalid bearer token, service account token is not valid yet]"

  • When running the timedatectl command from SSH to workload cluster nodes directly, the Local time is more than 2 minutes different than the RTC time, for example:

    # timedatectl

    Local time: Wed 2024-05-15 10:30:00 UTC
    Universal time: Wed 2024-05-15 10:30:00 UTC
    RTC time: Wed 2024-05-15 10:36:00 UTC
    Time zone: UTC (UTC, +0000)

    System clock synchronized: no

    NTP service: active

    RTC in local TZ: no

Cause

The authentication failures occur because the server from which users attempt to login is greater than 2 minutes in time ahead of the workload cluster node's local time. The token issued by Dex defaults to a 2 minute expiry, leading to a failure in authentication. If the source and destination server's times are greater than 2 minutes apart, the token can never be validated.

Resolution

  1. Configure an NTP server via ytt customization in plan-based TKGM clusters according to Legacy Cluster Configuration with ytt documentation
  2. Configure an NTP server via NTP_SERVER variable in class-based TKGM clusters according to Configuration File Variable Reference documentation
  3. For remediation of existing cluster nodes, Enable Periodic Time Synchronization in vSphere on all Control Plane and Worker nodes in the impacted workload cluster.