Secret Store Supervisor Service reports "Couldn't read secrets: permission denied"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• On the vSphere Client, the Secret Store Supervisor Service reports the status, "This service is not available", as shown in the following log snippet:

• In VMware Cloud Foundation Automation (VCFA), attempts to create a keyvaluesecret fail with a banner "Couldn't write secrets: permission denied"
• API aggregator logs from the Supervisor context report:
  
  kubectl -n svc-secret-store-domain-c## logs api-aggregator-59###db79-q##cn | less

INFO: [Timestamp] RestBaoStorage.go:234: Started 'Create' in Rest Provider..
INFO: [Timestamp] RestBaoStorage.go:163: Namespace is: svc-secret-store-domain-c##
INFO: [Timestamp] RestBaoStorage.go:281: URL:  http://##.##.##.##:8200/v1/secret/data/svc-secret-store-domain-c##/db-creds
INFO: [Timestamp] RestBaoStorage.go:534: Client token is empty, fetch new client token
INFO: [Timestamp] RestBaoStorage.go:294: Token expiration is {} [Timestamp] 
INFO: [Timestamp] RestBaoStorage.go:324: Method: , URL:  POST http://##.##.##.##:8200/v1/secret/data/svc-secret-store-domain-c##/db-creds
INFO: [Timestamp] RestBaoStorage.go:341: Request is Not GET. Setting content-type header..
[Timestamp] 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"Couldn't write secrets: permission denied"}: Couldn't write secrets: permission denied

• Execute into the secret-store pod and review the OpenBao initialization logs, which confirm a certificate verification failure during the Kubernetes authentication setup
  
  kubectl -n svc-secret-store-domain-c## exec -it secret-store-0 -- sh
cat /home/openbao/post-start.log

+ bao auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
+ echo 'Bao initialized and unsealed successfully!'
Bao initialized and unsealed successfully!
+ sed -n -e '/-.BEGIN/,/-.END/ p'
+ openssl s_client -showcerts -servername supervisor-cluster -connect ##.##.##.##:443
Connecting to ##.##.##.##
depth=0 OU=org-1.example.org, CN=kube-apiserver-domain-c##
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU=org-1.example.org, CN=kube-apiserver-domain-c##
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 OU=org-1.example.org, CN=kube-apiserver-domain-c##
verify return:1
DONE
+ sed 's/<ns-name>/svc-secret-store-domain-c##/g' /policy.hcl
+ bao policy write svc-secret-store-domain-c## -
+ cat /tmp/policy.hcl
Success! Uploaded policy: svc-secret-store-domain-c##
+ bao policy write api-aggregator -
+ cat /aggregator-policy.hcl
Success! Uploaded policy: api-aggregator
+ rm /tmp/policy.hcl
+ cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt /tmp/wcp.cert
+ cat /var/run/secrets/kubernetes.io/serviceaccount/token
+ cat /tmp/bundle.cert
+ bao write auth/kubernetes/config 'kubernetes_host=https://##.##.##.##:443' 

Cause

The OpenBao component (underlying engine for the Secret Store Service) is unable to communicate with the Kubernetes API server due to a Subject Alternative Name (SAN) mismatch on the kube-vip certificate.

This condition occurs in environments that transition from a Single-Node configuration to a High Availability (HA) configuration utilizing a Load Balancer. The pre-existing certificate does not contain the necessary SAN entries required for the new VIP endpoint, causing the Kubernetes authentication mount (auth/kubernetes/config) to fail verification.

Resolution

1. Replace the current VIP certificate with a self-signed or custom-signed certificate that includes the required SAN entries.
2. Reconfigure OpenBao Authentication:

Manually update the authentication configuration from the Supervisor context:

1. 1. Exec into secret-store pod:
      
      kubectl -n svc-secret-store-domain-c## exec -it secret-store-0 -- sh
      
      
   2. Export the new VIP server certificate:
      
openssl s_client -showcerts -servername supervisor-cluster -connect ##.##.##.##:443 </dev/null | sed -n -e '/-.BEGIN/,/-.END/ p' > /tmp/new-cert.cert
      
      
   3. Update the Kubernetes authentication configuration:

bao write auth/kubernetes/config \
token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
kubernetes_host=https://##.##.##.##:443 \
kubernetes_ca_cert=@/tmp/new-cert.cert
      
      
   4. Verify the login role:
      
bao write auth/kubernetes/login \
role=svc-secret-store-domain-c## \
jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token