CSI storage driver is unable to attach due to 403 errors for a VMware Cloud Director Container Service Extension pod

Products

• VMware Cloud Director

Issue/Introduction

• A VMware Cloud Director Container Service Extension pod is not coming up and the following error is occurring frequently:
  
  GRPC error: rpc error: code = Unknown desc = error while obtaining access token: [failed to refresh VCD client with the refresh token: [error getting bearer token: error authorizing service account...
  
  
• The CSI storage driver is unable to authenticate with the storage layer while attaching the PVC producing a FailedAttachVolume error.
• Attempts to connect to the VMware Cloud Director API via curl from the management or worker nodes either fails or produces an invalid response.

Environment

VMware Cloud Director Container Service Extension 4.2.3

Cause

The kubernetes nodes are unable to reach the VMware Cloud Director API to perform any actions, including the production of an access token.

Resolution

Configure the networking and firewall to permit access to the VMware Cloud Director API.