Secrets Service reports "Couldn't read secrets: permission denied"

Products

• VMware vSphere Kubernetes Service
• VCF Automation

Issue/Introduction

• On the vSphere Client, the Secret Store Supervisor Service reports the status, "This service is not available", as shown in the following log snippet:

• In VMware Cloud Foundation Automation (VCFA), viewing or creating a secret/key value secret fails with the banner "Couldn't write secrets: permission denied"
  
  
  

While connected to the Supervisor cluster context, the following symptoms are observed:

• Viewing API aggregator logs show:
  

  kubectl -n svc-secret-store-domain-c## logs api-aggregator-59###db79-q##cn | less
  
  INFO: [Timestamp] RestBaoStorage.go:234: Started 'Create' in Rest Provider..
  INFO: [Timestamp] RestBaoStorage.go:163: Namespace is: svc-secret-store-domain-c##
  INFO: [Timestamp] RestBaoStorage.go:281: URL:  http://##.##.##.##:8200/v1/secret/data/svc-secret-store-domain-c##/db-creds
  INFO: [Timestamp] RestBaoStorage.go:534: Client token is empty, fetch new client token
  INFO: [Timestamp] RestBaoStorage.go:294: Token expiration is {} [Timestamp] 
  INFO: [Timestamp] RestBaoStorage.go:324: Method: , URL:  POST http://##.##.##.##:8200/v1/secret/data/svc-secret-store-domain-c##/db-creds
  INFO: [Timestamp] RestBaoStorage.go:341: Request is Not GET. Setting content-type header..
  [Timestamp] 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"Couldn't write secrets: permission denied"}: Couldn't write secrets: permission denied

• Execute into the secret-store pod and review the OpenBao initialization logs, which confirm a certificate verification failure during the Kubernetes authentication setup or that the below certificate does not match the current Supervisor VIP certificate:
  

  kubectl -n svc-secret-store-domain-c10 exec -it secret-store-0 -- cat /home/openbao/post-start.log
  
  + bao auth enable kubernetes
  Success! Enabled kubernetes auth method at: kubernetes/
  + echo 'Bao initialized and unsealed successfully!'
  Bao initialized and unsealed successfully!
  + sed -n -e '/-.BEGIN/,/-.END/ p'
  + openssl s_client -showcerts -servername supervisor-cluster -connect ##.##.##.##:443
  Connecting to ##.##.##.##
  depth=0 OU=org-1.example.org, CN=kube-apiserver-domain-c##
  verify error:num=20:unable to get local issuer certificate
  verify return:1
  depth=0 OU=org-1.example.org, CN=kube-apiserver-domain-c##
  verify error:num=21:unable to verify the first certificate
  verify return:1
  depth=0 OU=org-1.example.org, CN=kube-apiserver-domain-c##
  verify return:1
  DONE

Environment

vSphere Supervisor

VMware Cloud Foundation 9.x

Cause

The OpenBao component (underlying engine for the Secret Store Service) is unable to communicate with the Kubernetes API server due to a mismatch with the Supervisor cluster's kube-vip certificate.

This condition can also occur in environments that transition from a Single-Node configuration to a High Availability (HA) configuration utilizing a Load Balancer. The pre-existing certificate does not contain the necessary SAN entries required for the new VIP endpoint, causing the Kubernetes authentication mount (auth/kubernetes/config) to fail verification.

Resolution

1. If necessary, replace the current Supervisor cluster's VIP certificate with a self-signed or custom-signed certificate that includes the required SAN entries.
   
   
2. Connect into the Supervisor cluster context
   
   
3. Restart the secret-store-0 pod to pick up the current Supervisor VIP certificate:
   1. Retrieve the namespace for secret-store:
      

      kubectl get ns | grep svc-secret-domain

      
      
      
   2. Perform the delete command on the pod which will automatically be recreated:
      

      kubectl -n svc-secret-domain-c## delete pod secret-store-0

       

4. Confirm that the restarted secret-store-0 pod is using the correct, current Supervisor VIP certificate:
   

   kubectl -n svc-secret-store-domain-c10 exec -it secret-store-0 -- bao read auth/kubernetes/config

   
   
   
5. Refresh and check that the permissions denied error is no longer present.
   
   

If the above restart does not reflect the correct Supervisor VIP certificate afterwards, the below steps can be used

1. 1. Exec into secret-store pod:
      
      

      kubectl -n svc-secret-store-domain-c## exec -it secret-store-0 -- sh

      
      
      
   2. Export the new VIP server certificate:
      


      openssl s_client -showcerts -servername supervisor-cluster -connect ##.##.##.##:443 </dev/null | sed -n -e '/-.BEGIN/,/-.END/ p' > /tmp/new-cert.cert

      
      
      
   3. Update the Kubernetes authentication configuration:

      bao write auth/kubernetes/config \
      token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
      kubernetes_host=https://##.##.##.##:443 \
      kubernetes_ca_cert=@/tmp/new-cert.cert

      
      
      
   4. Verify the login role:
      

      bao write auth/kubernetes/login \
      role=svc-secret-store-domain-c## \
      jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token