vSphere Client Namespace displays 0 Secrets and KeyValueSecret operations fail with 403 Permission Denied

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• vSphere Client Namespace displays 0 Secrets for all namespaces

.

• Attempts to create a new secret fail with the following error reported in the user interface:

• While connected to the Supervisor cluster context, the following symptoms are observed and the client token lease shows older date.

  Viewing API aggregator logs show:

  kubectl -n svc-secret-store-domain-<ID> logs api-aggregator-<POD_ID> | less

  INFO:keyvaluesecret_crd_rest.go:775: Method: ., URL: GET https://secret-store-service.svc-secret-store-domain-cx.svc.cluster.local:8200/v1/secret/metadata/secret-test?list=true
ERROR:keyvaluesecret_crd_rest.go:820: Bao request returned status code 403 with body {"errors":["permission denied"]}
ERROR:keyvaluesecret_crd_rest.go:332: Failed to list secrets in namespace secret-test: Couldn't list secrets: Bao request returned something other than 2xx
INFO:keyvaluesecret_crd_rest.go:314: LIST API called
INFO:keyvaluesecret_crd_rest.go:223: Namespace is: secret-test
INFO:keyvaluesecret_crd_rest.go:883: Client token lease {}, {} 0 xxxx-xx-xx 
INFO:11:21:30 keyvaluesecret_crd_rest.go:775: Method: ., URL: GET https://secret-store-service.svc-secret-store-domain-cx.svc.cluster.local:8200/v1/secret/metadata/secret-test?list=true
ERROR:11:21:30 keyvaluesecret_crd_rest.go:820: Bao request returned status code 403 with body {"errors":["permission denied"]}
ERROR:11:21:30 keyvaluesecret_crd_rest.go:332: Failed to list secrets in namespace secret-test: Couldn't list secrets: Bao request returned something other than 2xx

Environment

• VMware Cloud Foundation 9.x
• OpenBao v2.0.0
• secret-store 9.1.0

Cause

• This issue is caused by a logic error where the api-aggregator caches a Vault authentication token at startup and does not refresh it after expiration.
• All KeyValueSecret API calls fail with 403 Permission denied after the initial token expires, despite the credentials being correct.

Resolution

• A permanent resolution for this behavior is targeted for inclusion in the upcoming VMware Cloud Foundation (VCF) 9.1.2 release.
• To resolve this issue, apply one of the following workarounds:
  • Workaround 1: Restart the api-aggregator pod
    • Restarting the pod will force a re-authentication and clear the expired token. This must be repeated if the issue reoccurs.
      • Connect to the supervisor cluster.
      • Restart the deployment using the following command: kubectl rollout restart deployment api-aggregator -n <secret-store-service-namespace>

• • Workaround 2: Set a high Token TTL (Time-To-Live)
    • Extending the TTL delays the token expiration, acting as a long-term bypass.
      • SSH into the Vault/OpenBao pod:
        • kubectl -n <secret-store-namespace> exec -it secret-store-0 -- sh
      • Inside the pod, update the kubernetes authentication role with a high TTL (Example below sets a 30-day TTL and 90-day max TTL):
        • bao write auth/kubernetes/role/api-aggregator \
bound_service_account_names=api-aggregator \
bound_service_account_namespaces=<secret-store-service-namespace>\
ttl=720h \
max_ttl=2160h
      • Verify the changes were applied successfully:
        • bao read auth/kubernetes/role/api-aggregator
      • Exit the pod and restart the api-aggregator deployment to fetch the new token:
        • kubectl rollout restart deployment api-aggregator -n <secret-store-service-namespace>

Additional Information: If the API aggregator token is not expired but secret generation still fails, verify if the environment is impacted by related certificate trust issues as outlined in KB 415914.