vCenter 9.1 SSO login failure after upgrade with authentication error

Products

VCF Operations VMware Cloud Foundation

Issue/Introduction

VMware Cloud Foundation (VCF) Operations / vCenter Server Component: VCF Operations - Operate / vCenter SSO Symptoms: After upgrading vCenter Server to version 9.1, users are unable to log in via Single Sign-On (SSO). The browser redirects to an error page with the below message:
```
An error occurred during authentication. back to login screen
```

In the /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log, the following error is observed:

WARNING vsphere-ui 515 [vc@4413 threadName="agw-token-acq7" logger="com.vmware.vcenter.apigw.route.RoutingTable" 70000381 ###### 200021 RoutingTable ] Cannot route com.vmware.vcenter.authentication.token.null because vcenter ########-####-####-####-########### is not found
ERROR vsphere-ui 515 [vc@4413 threadName="agw-token-acq7" logger="com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider" 70000381 ###### 200021 AsyncTokenProvider ] Token acquisition failed for SSO domain vsphere.local(########-####-####-####-###########) com.vmware.vcenter.apigw.api.sso.tokenmgmt.TokenException: Failed to exchange JWT token for a SAML token for vcenterId 1#######-####-####-####-########### in domain vsphere.local(########-####-####-####-###########)
Caused by: com.vmware.vcenter.apigw.exception.VcenterIdNotFoundException: vCenter ID not found: '########-####-####-####-###########'.

ERROR vsphere-ui 515 [vc@4413 threadName="agw-token-####" logger="c.v.v.apigw.session.frontend.impl.FrontendSessionManagerImpl" ######## ###### 200021 FrontendSessionManagerImpl ] !!!!!!!!!! Failed to asynchronously acquire a local Act-As SAML token for front-end session '200021'. The session will be invalidated. java.util.concurrent.CompletionException: com.vmware.vcenter.apigw.api.sso.tokenmgmt.TokenException: Token acquisition failed for SSO domain vsphere.local(########-####-####-####-###########)
Caused by: com.vmware.vcenter.apigw.api.sso.tokenmgmt.TokenException: Token acquisition failed for SSO domain vsphere.local(########-####-####-####-###########)
Caused by: com.vmware.vcenter.apigw.exception.VcenterIdNotFoundException: vCenter ID not found: '########-####-####-####-############'.

ERROR vsphere-ui 2148 [vc@4413 threadName="http-nio-127.0.0.1-5090-exec-121" logger="com.vmware.vise.security.spring.DefaultAuthenticationProvider" 70008632 100149 200043 ] Unknown error during authentication com.vmware.vcenter.apigw.exception.ServiceClosedException: AsyncTokenProvider has been closed

In the /var/log/vmware/vsphere-ui/logs/apigw.log, the following error is observed:

Caused by: com. vmware. vcenter. apigw. exception. VcenterIdNotFoundException: vCenter ID not found: '########-####-####-####-###########'.

at com. vmware. vcenter. apigw. route.RoutingTable. tryRouteByVcId(Unknown Source)

at com. vmware. vcenter. apigw. route. RoutingTable. route(Unknown Source)

at com.vmware.vcenter.apigw.sso.tokenmgmt.impl. TokenExchangeServiceImpl$VcIdAwareApiProvider.invoke(Unknown Source)

at com. vmware.vapi.internal. bindings. Stub. invoke (Stub. java: 331)

at com.vmware.vapi.internal.bindings. Stub.invokeMethodAsync (Stub. java: 281)

at com. vmware. vapi. internal. bindings. Stub. invokeMethod (Stub. java : 178)

Environment

VMware Cloud Foundation 9.1
vIDB 9.1
vCenter Server 9.1

Cause

The issue is primarily caused by a synchronization failure between the vCenter Server and the Identity Broker following the 9.1 upgrade. The internal OAuth2 trust registration becomes stale, causing the AsyncTokenProvider to enter a closed state and the Suite Token to be missing from authorization requests.
This is also caused by stale/malformed cs.identity service registration entries in the vCenter Lookup Service (Lookupsvc) where the Node ID is missing.

Resolution

To resolve this either of the following can be used:

Option A: Using attached regen_csidentity.sh bash script:

Take an offline snapshot of the vCenter Server Appliance (VCSA)
Ensure SSH access is enabled and you have root credentials for the VCSA.
Upload or copy the attached regen_csidentity.sh script to the vCenter server.
Note: If you cannot connect via WinSCP, temporarily change the root shell:
```
chsh -s /bin/bash root

After upload, revert the shell:

chsh -s /bin/appliancesh root
```
Change to the directory of the script, and mark it as executable:
```
chmod +x regen_csidentity.sh
```
Run the script and follow the prompts:
Note: If running the script returns "bash: ./reset_machine_pw.sh: /bin/bash^M: bad interpreter: No such file or directory", run the following:
```
sed -i -e 's/\r$//' regen_csidentity.sh
```

The workflow of the script should look like the below:

root@vcenter [ ~ ]# ./regen_csidentity.sh
This script will regenerate the cs.identity service registration.
It is intended to be used when this service is missing the Node ID attribute.
See KB https://knowledge.broadcom.com/external/article/448977 for more details.
WARNING Node ID is missing from ########-####-####-####-###########!
WARNING Regeneration of cs.identity is required!
Would you like to proceed? (y/n): y

 What is needed prior and after to executing this fix?
 1: Offline snapshot or backup of the vCenter server.
 2: SSO Admin Password
 3: Restart all services afterward.

Have you taken snapshots/backups? (y/n): y

 INFO Snapshot/backup confirmed.
 INFO Continuing execution...
INFO prompting for password for administrator@<sso.domain>
🔐 Password for [email protected]:  ********************

INFO Unregistering old service(s)...
INFO Old services successfully removed.
INFO Creating new service...
INFO only one registration found.
INFO A restart of all services is required for the change to take effect.
Restart all services now? (y/n):
Proceeding...
Operation not cancellable. Please wait for it to finish...
Performing stop operation on profile: ALL...
...
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vmware-envoy-system-proxy...
...
root@vcenter [ ~ ]#

NOTE: If needed, there will be a regen_csidentity.log file in the same directory for troubleshooting.

Option B: Using lsdoctor for service re-registration

Take an offline snapshot of the vCenter Server Appliance (VCSA)
Ensure SSH access is enabled and you have root credentials for the VCSA.
Upload the latest lsdoctor ZIP file to the VCSA using a utility like WinSCP.
Change to the directory containing the file and run:
```
unzip lsdoctor.zip
cd lsdoctor-main
```
To rebuild service registrations execute the below command:
```
python lsdoctor.py -r
```
When prompted, select Option 2 (Replace all services with new services) to rebuild all registrations based on a template.
Template Matching: If the tool cannot find an exact template for your build, select the closest matching version from the provided list.
Restart All Services: To ensure all registrations are properly loaded, restart the full service stack:
```
service-control --stop --all && service-control --start --all 
```
Log in to the vSphere Client to verify accessibility and service health.

Option C: Manual steps for service re-registration:

Check if cs.identity has a node ID:

PNID=$(/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost)  && /usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://$HOSTNAME/lookupservice/sdk --type cs.identity | awk -v IGNORECASE=1 "/Service Type/,/$PNID/" | grep "Node ID" | awk '{print $3}'

If no value is returned, proceed to the next steps.

Set PNID and SSO domain variables:

PNID=$(/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost)

SSODOMAIN=$(/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost)

Get the service ID(s):

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://$HOSTNAME/lookupservice/sdk --type cs.identity | awk -v IGNORECASE=1 "/Service Type/,/$PNID/" | grep "Service ID" | awk '{print $3}'

For each service ID returned, unregister them one at a time (you will be prompted for the password):

/usr/lib/vmware-lookupsvc/tools/lstool.py unregister --url https://$HOSTNAME/lookupservice/sdk --user administrator@$SSODOMAIN --password "$(systemd-ask-password)" --id <service ID from step 2>

Create a new cs.identity service registration:

/usr/lib/vmware-sso/bin/sts-init-ls.sh --host-name $PNID --http-port 443

Restart all services:

service-control --stop --all && service-control --start --all

Attachments

regen_csidentity.sh get_app