vcf operations sddc manager tab displays authorization error unauthorized access

Products

• VCF Operations
• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When accessing the **SDDC Manager** tab within the **VCF Operations** console (formerly VMware Aria Operations), the following symptoms are observed: The UI displays an error message: 'Authorization Error: Unauthorized access'.
• The SDDC Manager dashboard fails to load, showing a continuous loading spinner.
• Standard troubleshooting steps, such as rebooting the SDDC Manager VM or increasing CPU/Memory resources, do not resolve the issue.

 

Environment

VCF 9.x

Cause

The The token_signing_key.json file, which stores the JSON Web Keys (JWK) used for secure inter-service communication between VCF Operations and SDDC Manager, has become corrupted or empty. This causes the SDDC Manager API to return an internal server error (500) when the VCF Adapter attempts to synchronize authorization tokens.

Resolution

1. Log in to the SDDC Manager VM via SSH as the 'vcf' user and switch to 'root'.
2. Navigate to the configuration directory: cd /etc/vmware/vcf/commonsvcs/
3. Rename the corrupted file to create a backup:
   mv token_signing_key.json token_signing_key.json.bak
   
   
4. Make sure the permissions of /etc/vmware/vcf/commonsvcs/ folder are correct - 750
   
   
   
   
5. Take SSH to VCF Operations Primary node, acquire the JWT to keys from VCF Operations
   1. curl -k  https://<VCF-Ops-Primary-fqdn>/suite-api/.well-known/openid-configuration
      
      Command Output
      
      {"issuer":"vcf_ops-#########-####-####-####-#############","jwks_uri":"https://<VCF-OPS-Primary-fqdn>/suite-api/jwks"}
      
      
   2. curl -k https://<VCF-OPS-Primary-fqdn>/suite-api/jwks
      
      Command Output
      
      {"keys":[{"kty":"RSA","e":"AQAB","kid":"####################################"}]}
      
      
      
6. Acquire the token with the below command
   
   token=$(curl -X POST -H "Content-Type: application/json" -d '{"username": "[email protected]","password": "<password>"}' --insecure  https://<sddc-manager>/v1/tokens | jq -r '.accessToken')
   
   
7. Publish the JWT keys acquired acqureid from 5a and 5b
   
   curl -k -X POST 'https://<sddc-manager>/v1/tokens/signing-keys' -H "Content-Type: application/json" -H "Authorization: Bearer $token" -d '{"issuer":"vcf_ops-#########-####-####-####-#############","keys":[{"kty":"RSA","e":"AQAB","kid":"####################################"}]}'


8. Validate that the signing key is pushed
   
   curl -X GET -H "Authorization: Bearer $token" --insecure https://<sddc-manager>/v1/tokens/signing-keys | json_pp
   
   
9. In the SDDC Manager, validate /etc/vmware/vcf/commonsvcs/token_signing_key.json has the JWT keys