Kubectl CLI fails with an error "per-RPC creds failed due to error" in TMC-SM
search cancel

Kubectl CLI fails with an error "per-RPC creds failed due to error" in TMC-SM

book

Article ID: 389066

calendar_today

Updated On:

Products

VMware Tanzu Mission Control - SM

Issue/Introduction

The Kubectl CLI fails with the following error in TMC-SM after nine hours from the initial login time. 

Error: rpc error: code = Internal desc = transport: per-RPC creds failed due to error: failed to run a local listener to facilitate login: listen tcp :80: bind: permission denied

Cause

Tanzu Mission Control Self-Managed manages user authentication using Pinniped Supervisor as the identity broker and requires an existing OIDC-compliant identity provider (IDP). Logging in using the Pinniped Supervisor via the Pinniped CLI is done using the OIDC authorization code flow.

The authorization code can then be exchanged using the Supervisor’s token endpoint for the following Supervisor-issued tokens, which are cached by the CLI in a file in the user’s home directory.

  • An ID token, which is valid for 2 minutes. This initial ID token is typically not used.
  • An opaque access token, which is valid for 2 minutes. This token will be used to fetch a new, cluster-scoped access token. It will only be sent back to the Supervisor, and never to any other server.
  • An opaque refresh token, which is valid for 9 hours from the time of the initial authentication. This token will be used to perform an OIDC refresh grant to fetch a new set of ID, access, and refresh tokens. It will only be sent back to the Supervisor, and never to any other server.

Note that none of the token or credential lifetimes described in this document are currently configurable. (One exception is the lifetime of ID tokens issued to OAuth2 clients created as OIDCClients may be configured by the administrator, but that does not apply when using the Pinniped CLI, which always uses the OAuth2 client called pinniped-cli.)

The maximum amount of time that any user can continue to refresh their Supervisor session is 9 hours from the initial login time. After that, the next refresh will fail and the user must perform a fresh login. This ensures that the user’s access privileges are updated at least once a day, even if the Supervisor cannot detect an access privilege change made in the external identity provider during the day.

Resolution

Delete TMC-SM context and recreate the context by following the below guideline.

# list the available contexts
tanzu context list
 
# replace the CONTEXT_NAME with the valid context, and accept yes for confirmation
tanzu context delete CONTEXT_NAME
 
# create tmc context with the valid token
tanzu tmc context create --endpoint tmcsm.example.com -i pinniped --basic-auth

Additional Information

More information about Pinnped tokens and credentials can be found in the following document,