Recreate WS1B OAuth2 Client if it gets deleted in ELM ring.

Products

• VMware vCenter Server

Issue/Introduction

This is to give users a way to recover from an issue where the WS1B OAuth2 Client can get deleted. The failure is triggered by a new VC joining an ELM ring that a WS1B based IDP configured.
When the OAuth2 client gets deleted, logins will fail with the error "oauth2.client with client.id.not.found"

Symptoms:
When we login to VC, we get an error:

In trustmanagement service logs we get this error

2023-08-22T05:57:10.173Z [tomcat-exec-4 [] ERROR com.vmware.vcenter.trustmanagement.vapi.impl.VcIdentityProvidersProviderImpl opId=] Error updating OAuth2 Client object: Failed to create oauth2 client with client ID <<client ID>> on host <<hostname>>

vc-ws1a-broker service logs will have this error:

2023-08-22T05:57:10,141 ERROR <hostname>:accesscontrol (acs-rds-db-ops) [CUSTOMER;f6d5060c-fed0-4498-afe0-2ceae61e17ce;127.0.0.1;29dfd191-a351-4d6a-a965-a9b5df48f828;-;-;-] com.vmware.vidm.accesscontrol.OAuth2ClientService - Failed to create client java.util.concurrent.CompletionException: org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.DataException: could not execute statement

Environment

VMware vCenter Server 8.0.1

Cause

The database field storing the redirect URIs has an upperlimit and we can hit this limit if we have a large number of nodes in ELM and the hostnames of the VCs are sufficiently long enough.

Resolution

Currently there is no workaround.

Workaround:

1. Collect required information

1.1. vCenter fully-qualified domain names (FQDNs)

Obtain direct root shell access to one of the vCenters in the environment and use the Lookup Service tool to get the list of vCenter FQDNs based on all known trustmanagement service endpoints:

python /usr/lib/vmware-lookupsvc/tools/lstool.py list \
  --url $(/usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost) \
  --product 'com.vmware.trustmanagement' \
  --type 'trustmanagement' \
  --ep-proto 'vapi.json.https' \
  --ep-type 'com.vmware.trustmanagement.vapi' \
  --as-spec 2>/dev/null | grep 'endpoint[0-9]*\.url'

Extract only the hostname portion of each of the resulting URLs (referred to below as <vc1-fqdn>, <vc2-fqdn>, ..., <vcn-fqdn>).

1.2. OAuth2 client ID and client secret

Call the vCenter Providers API to get the OAuth2 client ID and secret that were originally configured in VMware Identity Services.

First get a vCenter session ID to use in subsequent API calls (where <vc-local-user-name> is a vsphere.local user with administrative privileges):

curl -X POST -u <vc-local-user-name> 'http://localhost/rest/com/vmware/cis/session' | jq '.'

Copy and retain the 'value' field in the resulting JSON response for subsequent API calls (referred to below as <vc-session-id>).

Get vCenter identity provider information via the Providers API:

curl -X GET 'http://localhost/api/vcenter/identity/providers/customer' \
  --header 'vmware-api-session-id: <vc-session-id>' \
  --header 'Accept: application/json' | jq '.oidc | {client_id, client_secret}'

Extract the 'client_id' and 'client_secret' fields from the resulting JSON response (referred to below as <vc-provider-client-id> and <vc-provider-client-secret>).

2. Re-create the OAuth2 Client

First get an admin client token to access the VMware Identity Services OAuth2 Client API:

curl -X GET 'http://localhost/api/vcenter/identity/broker/tenants/customer/admin-client' \
  --header 'vmware-api-session-id: <vc-session-id>' \
  --header 'Accept: application/json' | jq '.'

Copy and retain the 'access_token' field in the resulting JSON response for subsequent API calls (referred to below as <admin-client-token>).

Call the VMware Identity Services API to create the OAuth2 Client with the information that was collected:

curl -X POST --cacert /var/lib/vmware/vmca/root.cer "https://$(hostname)/acs/t/customer/broker/oauth2-clients" \
  --header 'Authorization: HZN <admin-client-token>' \
  --header 'Content-Type: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.with.rule.sets+json' \
  --data '{
    "client_id": "<vc-provider-client-id>",
    "secret": "<vc-provider-client-secret>",
    "scope": [
        "openid",
        "profile",
        "user",
        "group"
    ],
    "access_token_ttl": 10080,
    "refresh_token_ttl": 525600,
    "refresh_token_idle_ttl": 525600,
    "grant_types": [
        "refresh_token",
        "client_credentials",
        "password",
        "authorization_code"
    ],
    "redirect_uris": [
        "https://<vc1-fqdn>/ui/login/oauth2/authcode",
        "https://<vc2-fqdn>/ui/login/oauth2/authcode",
        ...
        "https://<vcn-fqdn>/ui/login/oauth2/authcode"
    ],
    "post_logout_redirect_uris": [
        "https://<vc1-fqdn>/ui/login?logout",
        "https://<vc2-fqdn>/ui/login?logout",
        ...
        "https://<vcn-fqdn>/ui/login/oauth2/authcode"
    ]
}' | jq '.'

The resulting JSON response displays the details of the newly re-created OAuth2 Client.