API Error: Failed to list credentials [Service Unavailable] when configuring Velero Backup in VCF Automation

Products

• VCF Automation

Issue/Introduction

When attempting to enable Velero backup for Kubernetes clusters in a VMware Cloud Foundation (VCF) 9 environment, administrators encounter an error message when accessing: Manage & Govern > Kubernetes Management > Configurations > Credentials.

The UI displays the following banner: API Error: Failed to list credentials: [Service Unavailable: please try again later]

This prevents the creation or management of credentials required for Velero backup integration.

Environment

VCF Automation 9.0.x

Cause

The api-gateway component is unable to connect to the account-manager service via internal gRPC due to an expired TLS certificate. Although cert-manager may have successfully renewed the api-gateway-instack-tls secret, the running pods may not have reloaded the new certificate, causing an authentication handshake failure.

Resolution

To resolve this issue, verify the certificate status and force a rolling restart of the affected services to reload the valid certificates.

1. Verify the Secret Status: Check that the api-gateway-instack-tls secret contains a renewed certificate with a future expiration date

   kubectl get secret api-gateway-instack-tls -n prelude -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

    
2. Restart API Gateway: Force a rolling restart of the api-gateway-server deployment

   kubectl rollout restart deployment/api-gateway-server -n prelude
   kubectl rollout status deployment/api-gateway-server -n prelude

    
3. Restart Account Manager: Repeat the process for the account-manager-server to ensure it is also using the valid certificate (account-manager-server-tls)

   kubectl rollout restart deployment/account-manager-server -n prelude
   kubectl rollout status deployment/account-manager-server -n prelude

    
4. Restart API Gateway: Force a rolling restart of the  dataprotection-server to the certificate change to reflect.

   kubectl rollout restart deployment/dataprotection-server -n prelude
   kubectl rollout status deployment/dataprotection-server -n prelude

5. Verification: Once the rollouts are complete, verify the pods are running and the UI error has cleared

   kubectl get pods -n prelude -l app=api-gateway-server
   kubectl get pods -n prelude -l app=account-manager-server

    

Additional Information

This behavior will reoccur during future certificate rotations, including the next scheduled certificate renewal cycle. Should the issue occur again, restarting the affected services will allow to pick up the newly rotated certificates and restore normal functionality.