SDDC Manager UI error - Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift

Products

VMware SDDC Manager VMware Cloud Foundation VMware vCenter Server

Issue/Introduction

SDDC Manager > Workload domain > Update page has the error

Scheduling immediate update of bundle failed. Provided domain ID is invalid.
 
Retrieving configuration updates failed. Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift. Because configuration realized check failed on resources [88ac####-####-####-####-########b540]. Please check logs and fix the failures of drift configuration realized checks. Then restart the service to trigger the configuration realized checks again and re-try the API. If this does not resolve the issue, please contact GSS.

Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

DEBUG [vcf_dm,0ad########8f42,c2cb] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7200-exec-5]  Handler Error Response: {"errorCode":"FAILED_TO_FETCH_APPLICABLE_DRIFTS_CONFIGURATION_REALIZED_CHECK_FAILED","arguments":["WorkspaceOneBrokerConfigDrift","[88ac####-####-####-####-########b540]"],"message":"Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift. Because configuration realized check failed on resources [88ac####-####-####-####-########b540].","remediationMessage":"Please check logs and fix the failures of drift configuration realized checks. Then restart the service to trigger the configuration realized checks again and re-try the API. If this does not resolve the issue, please contact GSS.","referenceToken":"6####I"}
yyyy-mm-ddThh:mm:ss ERROR [vcf_dm,618########4085,4c2d] [c.v.v.c.o.a.ConfigDriftApiClient,dm-exec-7]  Failed to fetch drifts from owner service http://localhost/domainmanager
com.vmware.cloud.foundation.rest.configreconciler.runtime.ApiException:

ERROR [vcf_dm,0ad########8f42,c2cb] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7200-exec-5]  [2A4MU] Unable to compute applicability for drift {0}. Because configuration realized check failed on resources {1}. Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift. Because configuration realized check failed on resources [88ac####-####-####-####-########b540].
com.vmware.vcf.configreconciler.service.common.error.ConfigReconcilerException: Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift. Because configuration realized check failed on resources [88ac####-####-####-####-########b540].

Error in vCenter /var/log/vmware/trustmanagement/trustmanagement-svcs.log

[tomcat-exec-24 [] ERROR com.vmware.vcenter.trustmanagement.authbroker.BrokerClient  opId=] VMware Identity services unavailable
[tomcat-exec-24 [] ERROR com.vmware.vcenter.trustmanagement.vapi.impl.AdminClientProviderImpl  opId=] Error getting Auth broker CUSTOMER tenant's admin client token info due to VMware Identity services unavailable
com.vmware.vcenter.trustmanagement.authbroker.BrokerException: VMware Identity services unavailable

Caused by: com.vmware.vcenter.trustmanagement.authbroker.BrokerClient$HttpStatusException: API request GET_CLIENT_CREDENTIALS_TOKEN failed with response code 503 (Service Unavailable)

[tomcat-exec-16 [] INFO  com.vmware.vcenter.trustmanagement.authbroker.BrokerClient  opId=] API request GET_CLIENT_CREDENTIALS_TOKEN to url http://localhost:1080/external-vecs/http1/vcenter.example.com/443/acs/t/customer/token returned unexpected response code 400 and the following error information: {"error":"server_error","error_description":"Unable to generate the Token."}

[tomcat-exec-16 [] ERROR com.vmware.vcenter.trustmanagement.authbroker.BrokerClient  opId=] VMware Identity services unavailable
[tomcat-exec-16 [] ERROR com.vmware.vcenter.trustmanagement.vapi.impl.AdminClientProviderImpl  opId=] Error getting Auth broker CUSTOMER tenant's admin client token info due to VMware Identity services unavailable
com.vmware.vcenter.trustmanagement.authbroker.BrokerException: VMware Identity services unavailable


Caused by: com.vmware.vcenter.trustmanagement.authbroker.BrokerClient$HttpStatusException: API request GET_CLIENT_CREDENTIALS_TOKEN failed with response code 400 (Bad Request)

Environment

VMware Cloud Foundation 5.x

Cause

vc-ws1a-broker service configuration files get lost during RDU upgrade which leaves the vc-ws1a-broker service in a broken state. The drift check failure is caused by the VMware Identity services being unavailable.

Resolution

This is a workaround to be used by customers who did the RDU upgrade and currently have a non functional WS1 Broker service. This workaround script should be used only by customers who has WS1B service in the source machine (Ex : Any VC Release >=8.0u1)

Take offline snapshot of all vCenter servers in powered off state before executing the workaround steps.

Workaround:

Run the attached shell script to be able to configure an external Identity Provider.

Download the recover_ws1b.sh script attached to this KB and upload to vCenter using WinSCP
SSH to vCenter with root user

Get the operator client ID and secret from the masterapp credentials file

cat /var/lib/ws1/masterapp-credentials.json

If the file is not present you will see the below error

cat: /var/lib/ws1/masterapp-credentials.json: No such file or directory

Sample

root@vcsa01 [ ~ ]# cat /var/lib/ws1/masterapp-credentials.json
cat: /var/lib/ws1/masterapp-credentials.json: No such file or directory

To resolve this
1. Download Rotate_WS1B_secrets.sh attached to this KB and upload to vCenter server using WinSCP
2. Change the script as executable
```
chmod +x Rotate_WS1B_secrets.sh
```
3. Run the script on the VC to rotate WS1 broker service secrets
```
./Rotate_WS1B_secrets.sh
```
4. Get the operator client ID and secret from the masterapp credentials file and save it in a variable
```
cat /var/lib/ws1/masterapp-credentials.json
```
  Sample output
```
root@vcsa01 [ ~ ]# cat /var/lib/ws1/masterapp-credentials.json
{ "clientId": "operator_client_id", "clientSecret":"Bnv########################v4o=", "scopes":"admin system"}
```

Run the script as below

bash recover_ws1b.sh <SSO user> <SSO_Password> <clientSecret from Step # 3(a)>

In case the script fails to resolve the issue, then run it manually with steps below

Rotate WS1B secrets
1. Download the Rotate_WS1B_secrets.sh script attached to this KB and upload to vCenter using WinSCP.
2. SSH to vCenter with root user
3. Change the script as executable
```
chmod +x Rotate_WS1B_secrets.sh
```
4. Run the script on the VC to rotate WS1 broker service secrets
```
./Rotate_WS1B_secrets.sh
```

Get system token using the new operator_client_id and secret

Get the operator client ID and secret from the masterapp credentials file

cat /var/lib/ws1/masterapp-credentials.json

Sample output:

{ "clientId": "operator_client_id", "clientSecret":"Bnv########################v4o=", "scopes":"admin system"}

Call WS1B token endpoint to get a token with these credentials

curl -k -u 'operator_client_id:<OPERATOR CLIENT SECRET>' 'https://<VC HOSTNAME>/acs/t/HWS/token' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials'

Delete keys for tenants operator and customer

Using the token from step 3, call the following APIs to delete keys for customer and operator tenant

Delete keys for customer tenant

curl -k --location --request DELETE 'https://<VC HOSTNAME>/crypto/t/CUSTOMER/jwt/keys' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'

Delete keys for operator tenant

curl -k --location --request DELETE 'https://<VC HOSTNAME>/crypto/t/OPERATOR/jwt/keys' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'

To verify keys are deleted, run the following command

psql -U postgres VCDB -c "SELECT * FROM vidm_schema.\"Crypto_Keys\" WHERE \"keyContainer\" in ('CUSTOMER:JWT', 'OPERATOR:JWT');"

This should return zero rows.

Generate new keys for tenants operator and customer

Using the token from step 3, call the following APIs to generate new keys for customer and operator tenant

Generate new keys for customer tenant

curl -k --location --request POST 'https://<VC HOSTNAME>/crypto/t/CUSTOMER/jwt/keys?algorithmId=ALG_RSA_2048_SHA256' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'

Generate new keys for operator tenant

curl -k --location --request POST 'https://<VC HOSTNAME>/crypto/t/OPERATOR/jwt/keys?algorithmId=ALG_RSA_2048_SHA256' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'

Update HVC Settings store with new master appcredentials.
- Step 1 generates new master app credentials for WS1B service but the settings store still has the old credentials. To pick up the new credentials, follow these two steps
  1. Clear the credentials currently stored in settings store:
    1. First get a VC session
```
curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '<Admin user>:<Admin password>'
```
    2. Use sessionID from above step to clear settings store
```
curl -X PUT -k -H "vmware-api-session-id:<SESSION_ID>" http://localhost:1080/api/vcenter/settings/namespaces/ws1/items -d '{"items": {"master_client_secret": ""}}' -H "Content-Type: application/json"
```
  2. Restart Trustmanagement service
```
vmon-cli -r trustmanagement
```
Reset Client Secret in IDP object
- Call the Authbroker Update API to reset the External IDP Client ID. Use the same client ID that was used to setup the IDP before upgrade or generate a new client ID and use it here.
  
  Note: Client secret can be obtained from the Okta server.
  Log into Okta server as Admin, search for the Oauth2 app created for the VC by clientID.
  We can recover the existing secret from under "CLIENT SECRETS" or generate a new one
  - First get a VC session
```
curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '<Admin user>:<Admin password>'
```
  - Call Authbroker API LIST to get IDP ID
```
curl -k --location --request GET 'https://localhost/api/vcenter/identity/authbrokeridp' --header 'vmware-api-session-id: <VC SESSION ID>' | jq
```
    Note IPD ID of the IDP configured with customer tenant
  - Call Authbroker API Update to reset the client secret
```
curl -k --location --request PATCH 'https://localhost/api/vcenter/identity/authbrokeridp/<IDP ID>' --header 'vmware-api-session-id: <VC SESSION ID>' --header 'Content-Type: application/json' --data-raw '{"client_secret" : "<CLIENT SECRET>"}'
```
Restart Services
1. Restart WS1B service
```
vmon-cli -r vc-ws1a-broker
```
2. Restart Trustmanagement service
```
vmon-cli -r trustmanagement
```
3. Restart STS service
```
vmon-cli -r sts
```

Verify everything works

Verify we are able to generate admin client tokens, generate a VC session and call the below APIs

First get a VC session

curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '<Admin user>:<Admin password>'

Generate admin client token for CUSTOMER tenant

curl -k --location --request GET 'https://localhost/api/vcenter/identity/broker/tenants/CUSTOMER/admin-client' --header 'vmware-api-session-id: <VC SESSION ID>'

Generate admin client token for OPERATOR tenant

curl -k --location --request GET 'https://localhost/api/vcenter/identity/broker/tenants/OPERATOR/admin-client' --header 'vmware-api-session-id: <VC SESSION ID>'

Perform a password grant login with a user from the external domain

curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '[email protected]:<password>'

Login from the VC UI with a user from external domain

Attachments

Rotate_WS1B_secrets.sh get_app

recover_ws1b.sh get_app