SDDC Manager UI error - Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift
search cancel

SDDC Manager UI error - Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift

book

Article ID: 377519

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • SDDC Manager > Workload domain > Update page has the error

    Scheduling immediate update of bundle failed. Provided domain ID is invalid.
 
Retrieving configuration updates failed. Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift. Because configuration realized check failed on resources [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]. Please check logs and fix the failures of drift configuration realized checks. Then restart the service to trigger the configuration realized checks again and re-try the API. If this does not resolve the issue, please contact GSS.

     

  • Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

    yyyy-mm-ddThh:mm:ss DEBUG [vcf_dm,0adf714c25328f42,c2cb] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7200-exec-5]  Handler Error Response: {"errorCode":"FAILED_TO_FETCH_APPLICABLE_DRIFTS_CONFIGURATION_REALIZED_CHECK_FAILED","arguments":["WorkspaceOneBrokerConfigDrift","[xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]"],"message":"Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift. Because configuration realized check failed on resources [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx].","remediationMessage":"Please check logs and fix the failures of drift configuration realized checks. Then restart the service to trigger the configuration realized checks again and re-try the API. If this does not resolve the issue, please contact GSS.","referenceToken":"aaaaa"}
yyyy-mm-ddThh:mm:ss ERROR [vcf_dm,618d6878dbc44085,4c2d] [c.v.v.c.o.a.ConfigDriftApiClient,dm-exec-7]  Failed to fetch drifts from owner service http://localhost/domainmanager
com.vmware.cloud.foundation.rest.configreconciler.runtime.ApiException:

yyyy-mm-ddThh:mm:ss ERROR [vcf_dm,0adf714c25328f42,c2cb] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7200-exec-5]  [2A4MU] Unable to compute applicability for drift {0}. Because configuration realized check failed on resources {1}. Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift. Because configuration realized check failed on resources [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx].
com.vmware.vcf.configreconciler.service.common.error.ConfigReconcilerException: Unable to compute applicability for drift WorkspaceOneBrokerConfigDrift. Because configuration realized check failed on resources [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx].

     

  • Error in vCenter /var/log/vmware/trustmanagement/trustmanagement-svcs.log

    yyyy-mm-ddThh:mm:ss [tomcat-exec-24 [] ERROR com.vmware.vcenter.trustmanagement.authbroker.BrokerClient  opId=] VMware Identity services unavailable
yyyy-mm-ddThh:mm:ss [tomcat-exec-24 [] ERROR com.vmware.vcenter.trustmanagement.vapi.impl.AdminClientProviderImpl  opId=] Error getting Auth broker CUSTOMER tenant's admin client token info due to VMware Identity services unavailable
com.vmware.vcenter.trustmanagement.authbroker.BrokerException: VMware Identity services unavailable

Caused by: com.vmware.vcenter.trustmanagement.authbroker.BrokerClient$HttpStatusException: API request GET_CLIENT_CREDENTIALS_TOKEN failed with response code 503 (Service Unavailable)

Environment

VMware Cloud Foundation 5.x

Cause

vc-ws1a-broker service configuration files get lost during RDU upgrade which leaves the vc-ws1a-broker service in a broken state. The drift check failure is caused by the VMware Identity services being unavailable.

Resolution

This is a workaround to be used by customers who did the RDU upgrade and currently have a non functional WS1 Broker service.  This workaround script should be used only by customers who has WS1B service in the source machine (Ex : Any VC Release >=8.0u1)

Take offline snapshot of all vCenter servers in powered off state before executing the workaround steps.

Workaround:

Run the attached shell script to be able to configure an external Identity Provider.

  1. Download the recover_ws1b.sh script attached to this KB and upload to vCenter using WinSCP
  2. SSH to vCenter with root user
  3. Get the operator client ID and secret from the masterapp credentials file 
    cat /var/lib/ws1/masterapp-credentials.json
  4. Run the script as below 
    bash recover_ws1b.sh <Admin user> <Admin password> <External IDP Client secret from Step # 3>

Incase the script fails to resolve the issue then run it manually with below steps

  1. Rotate WS1B secrets
    1. Download the Rotate_WS1B_secrets.sh script attached to this KB and upload to vCenter using WinSCP.
    2. SSH to vCenter with root user
    3. Change the script as executable 
      chmod +x Rotate_WS1B_secrets.sh
    4. Run the script on the VC to rotate WS1 broker service secrets 
      /Rotate_WS1B_secrets.sh
  2. Get system token using the new operator_client_id and secret
    1. Get the operator client ID and secret from the masterapp credentials file 
      cat /var/lib/ws1/masterapp-credentials.json
      Sample output:
      { "clientId": "operator_client_id", "clientSecret":"ZLt0#######nOk=", "scopes":"admin system"}\
    2. Call WS1B token endpoint to get a token with these credentials
      curl -k -u 'operator_client_id:<OPERATOR CLIENT SECRET>' 'https://<VC HOSTNAME>/acs/t/HWS/token' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials'
  3. Delete keys for tenants operator and customer
    • Using the token from step 3, call the following APIs to delete keys for customer and operator tenant
      • Delete keys for customer tenant 
        curl -k --location --request DELETE 'https://<VC HOSTNAME>/crypto/t/CUSTOMER/jwt/keys' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'
      • Delete keys for operator tenant 
        curl -k --location --request DELETE 'https://<VC HOSTNAME>/crypto/t/OPERATOR/jwt/keys' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'
      • To verify keys are deleted, run the following command 
        psql -U postgres VCDB -c "SELECT * FROM vidm_schema.\"Crypto_Keys\" WHERE \"keyContainer\" in ('CUSTOMER:JWT', 'OPERATOR:JWT');"
        This should return zero rows.

  4. Generate new keys for tenants operator and customer
    • Using the token from step 3, call the following APIs to generate new keys for customer and operator tenant
      • Generate new keys for customer tenant 
        curl -k --location --request POST 'https://<VC HOSTNAME>/crypto/t/CUSTOMER/jwt/keys?algorithmId=ALG_RSA_2048_SHA256' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'
      • Generate new keys for operator tenant 
        curl -k --location --request POST 'https://<VC HOSTNAME>/crypto/t/OPERATOR/jwt/keys?algorithmId=ALG_RSA_2048_SHA256' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'


  5. Update HVC Settings store with new master appcredentials.
    • Step 1 generates new master app credentials for WS1B service but the settings store still has the old credentials. To pick up the new credentials, follow these two steps
      1. Clear the credentials currently stored in settings store:
        1. First get a VC session 
          curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '<Admin user>:<Admin password>'
        2. Use sessionID from above step to clear settings store 
          curl -X PUT -k -H "vmware-api-session-id:<SESSION_ID>" http://localhost:1080/api/vcenter/settings/namespaces/ws1/items -d '{"items": {"master_client_secret": ""}}' -H "Content-Type: application/json"
      2. Restart Trustmanagement service 
        vmon-cli -r trustmanagement
  6. Reset Client Secret in IDP object
    • Call the Authbroker Update API to reset the External IDP Client ID. Use the same client ID that was used to setup the IDP before upgrade or generate a new client ID and use it here.

      Note: Client secret can be obtained from the Okta server.
      Log into Okta server as Admin, search for the Oauth2 app created for the VC by clientID.
      We can recover the existing secret from under "CLIENT SECRETS" or generate a new one
      • First get a VC session 
        curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '<Admin user>:<Admin password>'
      • Call Authbroker API LIST to get IDP ID 
        curl -k --location --request GET 'https://localhost/api/vcenter/identity/authbrokeridp' --header 'vmware-api-session-id: <VC SESSION ID>' | jq

        Note IPD ID of the IDP configured with customer tenant
      • Call Authbroker API Update to reset the client secret 
        curl -k --location --request PATCH 'https://localhost/api/vcenter/identity/authbrokeridp/<IDP ID>' --header 'vmware-api-session-id: <VC SESSION ID>' --header 'Content-Type: application/json' --data-raw '{"client_secret" : "<CLIENT SECRET>"}'
  7. Restart Services
    1. Restart WS1B service 
      vmon-cli -r vc-ws1a-broker
    2. Restart Trustmanagement service 
      vmon-cli -r trustmanagement
    3. Restart STS service 
      vmon-cli -r sts
  8. Verify everything works
    1. Verify we are able to generate admin client tokens, generate a VC session and call the below APIs
      • First get a VC session 
        curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '<Admin user>:<Admin password>'
      • Generate admin client token for CUSTOMER tenant 
        curl -k --location --request GET 'https://localhost/api/vcenter/identity/broker/tenants/CUSTOMER/admin-client' --header 'vmware-api-session-id: <VC SESSION ID>'
      • Generate admin client token for OPERATOR tenant 
        curl -k --location --request GET 'https://localhost/api/vcenter/identity/broker/tenants/OPERATOR/admin-client' --header 'vmware-api-session-id: <VC SESSION ID>'
    2. Perform a password grant login with a user from the external domain 
      curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '[email protected]:<password>'
  9. Login from the VC UI with a user from external domain

Attachments

Rotate_WS1B_secrets.sh get_app
recover_ws1b.sh get_app
Powered by Wolken Software