Error "Invalid issuer in token. JWT issuer is absent in token or does not match supported token issuers. Retry with a fresh token."

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Any VCF Fleet LCM service that validates JWTs issued by the OPS (VCF Operations) trusted issuer can intermittently fail to validate after the VCF Ops certificate is rotated.

The following symptoms are typically observed together on whichever affected service is involved:

Failed to initialize STSAdapter: componentType=ops with a TLS handshake error certificate_unknown(46) when connecting to https://<ops-fqdn>/suite-api/.well-known/openid-configuration.

Environment

VCF 9.0.1

Cause

• The OPS trusted-issuer connection uses certificate pinning at initialization of fleet pods making it so it trusts only the single certificate supplied at that time by VSP Issuers.
• When IMDS serves a stale certificate for OPS and one of the fleet pods is restarted, the platform's existing certificate-retrust mechanism runs and reports success, but it updates a different trust store than the one the pinned connection, so the mismatch is never resolved by that automatic mechanism. The adapter remains broken until IMDS itself reports the corrected certificate at the moment the adapter is (re)initialized.

Resolution

Note: You need to be in the root account to execute the commands below.

1. Connect to one of the VCF service control planes and fetch the FQDN by running the following command:

   kubectl get pd -n vmsp-platform -ojson | jq | grep fqdn

2. Set the VSP_HOST variable with the FQDN value:

   VSP_HOST=my.vcf-mgmt.domain

3. Run the following command to fetch the token:

   curl -ks --request POST \
     --url https://$VSP_HOST/api/v1/identity/token \
     --header 'Content-Type: application/x-www-form-urlencoded' \
     --data grant_type=password \
     --data '[email protected]' \
     --data 'password=<password>'

   Note: Replace <password> with the actual VCF service password.

4. Set the TOKEN environment variable with the access_token value from the token HTTP response:

   

   Copy the access_token value and update to create the variable:

   TOKEN="<access_token>"

5. Get the current OPS trusted issuer:

   curl -sk -X GET https://${VSP_HOST}/api/v1/identity/trusted-issuers/ops \
     -H "Authorization: Bearer ${VSP_TOKEN}" \
     -H 'Accept: application/json' -o /tmp/ops-trusted-issuer.json

6. Fetch the OPS actual live certificate and save it to /tmp/ops-cert.pem:

   openssl s_client -connect <ops-fqdn>:443 -servername <ops-fqdn> </dev/null 2>/dev/null | openssl x509 > /tmp/ops-cert.pem

7. Create another variable for the certificate NEW_CERT:

   NEW_CERT=$(jq -Rs . /tmp/ops-cert.pem)

8. Run the following command to split the trusted certificate:

   jq --argjson newcert "${NEW_CERT}" '.nodes[0].certificates[0] = $newcert' \
     /tmp/ops-trusted-issuer.json > /tmp/ops-trusted-issuer-updated.json

9. Run the following command to update the trusted certificate:

   curl -sk -w '\nHTTP_STATUS:%{http_code}\n' \
     -X PUT https://<VSP_HOST>/api/v1/identity/trusted-issuers/ops \
     -H "Authorization: Bearer ${VSP_TOKEN}" \
     -H 'Content-Type: application/json' \
     -H 'Accept: application/json' \
     -d @/tmp/ops-trusted-issuer-updated.json

10. Log out and log in to the VCF operation page. The authentication error should now be resolved.

Additional Information

1. Command to inspect the live certificate: openssl s_client -connect <ops-fqdn>:443 -servername <ops-fqdn> </dev/null 2>/dev/null | openssl x509 -noout -text
2. Command to inspect the raw IMDS-facing data: kubectl -n vmsp-platform get secret identity-trusted-issuers -o jsonpath='{.data.issuers\.json}' | base64 -d | jq .