UI unavailable in VCF Operations due to key rotation

Products

• VCF Operations

Issue/Introduction

Customers or administrators may encounter the following indicators when the system is in a "Not Ready" state:

• Service Status: The VCF Management pane displays a "Not Ready" status in the management console.

• Permission Errors (403 Forbidden): While interacting with the interface, the system fails to fetch status updates.

 

• • Open your browser's Developer Tools (F12) and navigate to the Network tab.

  • Refresh the page or attempt to trigger the status update.

  • Look for failures in the status API. You will observe a 403 Forbidden response code.

 

 

Reference Log: /var/log/nginx/access.log

"GET /lcm/bootstrap/api/status HTTP/1.1" 403

Environment

VCF Operations 9.0.x with VCF Operations fleet management appliance 9.0.x

Cause

This issue stems from a key rotation mismatch, which interrupts the establishment of secure UI sessions.

Resolution

To resolve the "Not Ready" status, follow the troubleshooting steps below. Start with Option 1. If the issue persists, proceed to the  Option 2.

Option 1: Restart the Lifecycle Service

This is the least disruptive step and should be attempted first to refresh the service state.

1. SSH into the VCF Management appliance using root credentials.

2. Restart the lifecycle server service by executing the following command.

   systemctl restart vrlcm-server.service
   

3. Verify if the "status" API failures (403 errors) have resolved in the UI.

 

---

Option 2: Reset Signing Keys in the Database

If a service restart does not resolve the 403 Forbidden errors, the signing keys in the database may be out of sync and require a reset.

Warning: Before proceeding with database modifications, ensure you have a fresh snapshot or backup of the VCF Operations nodes.

1. Log out of the VCF Operations user interface.

2. SSH into the VCF Operations Master or Replica node.

3. Access the Postgres database by running the following command:

   su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d vcopsdb -p 5433"
   

4. Clear the existing signing keys by executing the following SQL statement:

   DELETE FROM kv_ops_signing_keys;
   

5. Exit the database prompt (type \q or exit).

6. SSH back into the VCF Management appliance.

7. Restart the lifecycle service once more to regenerate the keys:

   systemctl restart vrlcm-server.service