When accessing the SDDC Manager tab or navigating to Fleet Management an 'Unauthorized access message' is seen in VCF Operations.
search cancel

When accessing the SDDC Manager tab or navigating to Fleet Management an 'Unauthorized access message' is seen in VCF Operations.

book

Article ID: 435716

calendar_today

Updated On:

Products

VCF Operations VCF Operations/Automation (formerly VMware Aria Suite) VMware vSphere ESXi

Issue/Introduction

When accessing the SDDC Manager tab within the VCF Operations UI, or navigating to Fleet Management > Lifecycle, you observe the following:

  • The UI displays an error message: Authorization Error: Unauthorized access.
  • The SDDC Manager dashboard fails to load, showing a continuous loading spinner.
  • The issue persists even after rebooting the SDDC Manager VM or increasing resources.
  • The SDDC Manager API returns an internal server error (500) during token synchronization.

Environment

VCF Operations 9.x

Cause

  • The token_signing_key.json file, located in /etc/vmware/vcf/commonsvcs/ on the SDDC Manager, has become corrupted or is 0-bytes.
  • This file stores the JSON Web Keys (JWK) necessary for secure communication between VCF Operations and SDDC Manager. When corrupted, the authentication handshake fails.

Resolution

To manually resolve the authorization error in existing environments, follow these steps to push valid keys from VCF Operations to the SDDC Manager:

Step 1: Prepare the SDDC Manager Environment

  1. Log in to the SDDC Manager VM via SSH as the vcf user and switch to root.
  2. Navigate to the configuration directory: 

    cd /etc/vmware/vcf/commonsvcs/
  3. Create a backup of the corrupted file: 

    mv token_signing_key.json token_signing_key.json.bak

  4. Verify the directory permissions are set to 750: 

    chmod 750 /etc/vmware/vcf/commonsvcs/


Step 2: Retrieve Keys from VCF Operations

  1. Log in to the VCF Operations Primary node via SSH.
  2. Retrieve the JWT Issuer string: 

    curl -k https://<VCF_OPS_FQDN>/suite-api/.well-known/openid-configuration 

    Command Output

    {"issuer":"vcf_ops-#########-####-####-####-#############","jwks_uri":"https://<VCF-OPS-Primary-fqdn>/suite-api/jwks"}

  3. Retrieve the JWT Keys payload: 

    curl -k https://<VCF_OPS_FQDN>/suite-api/jwks                (Note the "keys" array in the output)

    Command Output

    {"keys":[{"kty":"RSA","e":"AQAB","kid":"####################################"}]}

Step 3: Publish Keys to SDDC Manager

  1. Generate an API authorization token on the SDDC Manager CLI session  (replace <SSO_PASSWORD> and <SDDC_MGR_FQDN>): 

    token=$(curl -X POST -H "Content-Type: application/json" -d '{"username": "[email protected]","password": "<SSO_PASSWORD>"}' --insecure https://<SDDC_MGR_FQDN>/v1/tokens | jq -r '.accessToken')

  2. Post the keys to the SDDC Manager signing-keys endpoint using the issuer and keys gathered in Step 2:

    curl -k -X POST 'https://<sddc-manager>/v1/tokens/signing-keys' -H "Content-Type: application/json" -H "Authorization: Bearer $token" -d '{"issuer":"vcf_ops-#########-####-####-####-#############","keys":[{"kty":"RSA","e":"AQAB","kid":"####################################"}]}'

Step 4: Validation

  1. Confirm the keys are active via the API: 

    curl -X GET -H "Authorization: Bearer $token" --insecure https://<SDDC_MGR_FQDN>/v1/tokens/signing-keys | json_pp
  2. Verify that /etc/vmware/vcf/commonsvcs/token_signing_key.json has been automatically recreated and populated on the SDDC Manager filesystem.