Error "Invalid issuer in token. JWT issuer is absent in token or does not match supported token issuers. Retry with a fresh token."

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Any VCF Fleet LCM service that validates JWTs issued by the OPS (VCF Operations) trusted issuer can intermittently fail to validate after the VCF Ops certificate is rotated.

The following symptoms are typically observed together on whichever affected service is involved:

Failed to initialize STSAdapter: componentType=ops with a TLS handshake error certificate_unknown(46) when connecting to https://<ops-fqdn>/suite-api/.well-known/openid-configuration.

Environment

VCF 9.0.1

Cause

• The OPS trusted-issuer connection uses certificate pinning at initialization of fleet pods making it so it trusts only the single certificate supplied at that time by VSP Issuers.
• When IMDS serves a stale certificate for OPS and one of the fleet pods is restarted, the platform's existing certificate-retrust mechanism runs and reports success, but it updates a different trust store than the one the pinned connection, so the mismatch is never resolved by that automatic mechanism. The adapter remains broken until IMDS itself reports the corrected certificate at the moment the adapter is (re)initialized.

Resolution

Note: You need to be in root account to execute below commands

1. Connect to one of VCF service control plan and then fetch FQDN by running below command:
   

   kubectl get pd -n vmsp-platform -ojson | jq | grep fqdn

2. Set the VSP_HOST variable with FQDN value:
   

   VSP_HOST=my.vcf-mgmt.domain

3. Run below command to fetch token:
   

   curl -ks --request POST \
     --url https://$VSP_HOST/api/v1/identity/token \
     --header 'Content-Type: application/x-www-form-urlencoded' \
     --data grant_type=password \
     --data '[email protected]' \
     --data 'password=<password>' 

   Note: <password>, replace it with actual VCF service password.

4. Set TOKEN environment variable with the `access_token` value from token HTTP response:
   
   

   Copy the access_token value  and update to create variable:
   
   TOKEN="<access_token>"

5. Get current OPS trusted issuer
   

   curl -sk -X GET https://${VSP_HOST}/api/v1/identity/trusted-issuers/ops \
     -H "Authorization: Bearer ${VSP_TOKEN}" \
     -H 'Accept: application/json' -o /tmp/ops-trusted-issuer.json

6. Fetch OPS actual live certificate and save to the file as /tmp/ops-cert.pem
   

   openssl s_client -connect <ops-fqdn>:443 -servername <ops-fqdn> </dev/null 2>/dev/null | openssl x509 > /tmp/ops-cert.pem

7. Create another variable for cert "NEW_CERT"
   

   NEW_CERT=$(jq -Rs . /tmp/ops-cert.pem)

8. Run below command to split the trusted certificate:
   

   jq --argjson newcert "${NEW_CERT}" '.nodes[0].certificates[0] = $newcert' \
     /tmp/ops-trusted-issuer.json > /tmp/ops-trusted-issuer-updated.json

9. Run below command to update the trusted certificate:
   

   curl -sk -w '\nHTTP_STATUS:%{http_code}\n' \
     -X PUT https://<VSP_HOST>/api/v1/identity/trusted-issuers/ops \
     -H "Authorization: Bearer ${VSP_TOKEN}" \
     -H 'Content-Type: application/json' \
     -H 'Accept: application/json' \
     -d @/tmp/ops-trusted-issuer-updated.json

10. Logout and login to VCF operation page and Authentication Error should be resolved.