Upgraded Workload Domains fail to join Embedded Identity Broker in VCF 9.x: "VMware Identity services unavailable"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Unable to join specific vCenter to the Embedded Identity Broker (VCF SSO) via Fleet Management.
• This issue exclusively affects vCenters that have been upgraded from version 8.x to 9.x. New deployments of vCenter Server 9.x configure successfully and do not exhibit this behavior.
• The following log snippets are observed within the vCenter Server logs: 
  
  /var/log/vmware/vc-ws1a-broker/federation-service.log:
YYYY-MM-DDTHH:MM:SS,353 WARN  <REDACTED_HOSTNAME>:federation (ForkJoinPool-2-worker-1) [-;-;-;-;-;-] com.vmware.vidm.common.gateway.mesh.GatewayAuthProvider - Failed to acquire token, returning cached token - Optional.empty, ScUnauthorizedException[Operation: POST -> http://localhost:10114/acs/token][Status:401]

/var/log/vmware/trustmanagement/trustmanagement-svcs.log
YYYY-MM-DDTHH:MM:SS.186Z [tomcat-exec-22 [] ERROR com.vmware.vcenter.trustmanagement.authbroker.BrokerClient  opId=] VMware Identity services unavailable
    defaultMessage = VMware Identity services unavailable,
    args = [VMware Identity services unavailable],
    defaultMessage = VMware Identity services unavailable,
    args = [VMware Identity services unavailable],


• Generating admin client tokens using VC session APIs fails. This can be validated as follows : 
  
  First get a vCenter session :
  curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '<Admin user>:<Admin password>'



  Generate admin client token for CUSTOMER tenant :
  curl -k --location --request GET 'https://localhost/api/vcenter/identity/broker/tenants/CUSTOMER/admin-client' --header 'vmware-api-session-id: <VC SESSION ID>'

  Generate admin client token for OPERATOR tenant : 
  curl -k --location --request GET 'https://localhost/api/vcenter/identity/broker/tenants/OPERATOR/admin-client' --header 'vmware-api-session-id: <VC SESSION ID>'

Environment

VMware Cloud Foundation 9.x
VMware vCenter

Cause

The issue is caused by the vc-ws1a-broker service entering an unhealthy state. Potential causes include:

• The vc-ws1a-broker service configuration files failing to migrate or being inadvertently purged during the data transfer phase, resulting in a broken or uninitialized service state.
• Inadequate file permissions or incorrect ownership mappings preventing the vc-ws1a-broker service account from accessing necessary operational directories or migrated database schema.

Resolution

Before proceeding, ensure to have a valid snapshot of the vCenter Server. Additionally, perform a file-based backup of the vCenter Server using the vCenter Server Appliance Management Interface (VAMI).

1. Download the recover_ws1b.sh script attached to this KB article and upload it to the vCenter Server using a secure transfer utility.
   
   
2. Establish an SSH session to the vCenter Server and authenticate as the root user.
   
   
3. Retrieve the operator client ID and secret from the masterapp credentials file by executing the following command:
   cat /var/lib/ws1/masterapp-credentials.json
   
   
4. Evaluate the output of the above command.
   If the file is present: Note the <clientSecret> value and proceed directly to Step 5.
   If the file is missing: following error is observed (e.g., cat: /var/lib/ws1/masterapp-credentials.json: No such file or directory). Perform the following sub-steps to resolve:
   1. Download the Rotate_WS1B_secrets.sh script attached to this KB article and upload it to the vCenter Server.
   2. Modify the script permissions to make it executable:
      chmod +x Rotate_WS1B_secrets.s
   3. Convert Windows carriage returns (CRLF) to Unix line feeds (LF):
      sed -i -e 's/\r//g' Rotate_WS1B_secrets.sh
   4. Execute the script to rotate the Workspace ONE Broker service secrets:
      ./Rotate_WS1B_secrets.sh
   5. Retrieve the operator client ID and secret from the newly generated masterapp credentials file and note the clientSecret value:
      cat /var/lib/ws1/masterapp-credentials.json
   6. Sample Output:
      { "clientId": "operator_client_id", "clientSecret":"Bnv########################v4o=", "scopes":"admin system"}
      
      
5. Execute the recovery script using your specific SSO credentials and the clientSecret obtained in the previous steps:
   bash recover_ws1b.sh <SSO_User> <SSO_Password> <clientSecret>
   
   
6. Restart the vmware-trustmanagement service to apply the changes:
   service-control --stop vmware-trustmanagement
service-control --start vmware-trustmanagement