vCenter Microsoft Entra ID provider change fails with error "Failed to get an access token on host <VC_FQDN> for tenant HWS"
search cancel

vCenter Microsoft Entra ID provider change fails with error "Failed to get an access token on host <VC_FQDN> for tenant HWS"

book

Article ID: 437622

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

In the vCenter UI page, change of provider to Microsoft Entra ID fails with the below error :

 

 

1. /var/log/vmware/trustmanagement/trustmanagement-svcs.log 
 
YYYY-MM-DDT06:57:13.974Z [main [] ERROR com.vmware.sync.subscribe.SusbscribeDbStore  opId=] SusbscribeDbStore intialized with dbUserName: trustmanagement_sync_db 
YYYY-MM-DDT06:57:56.707Z [ws1bTLSCertExpiryCheckerScheduler-1 [] ERROR com.vmware.vcenter.trustmanagement.authbroker.TenantInitializer  opId=] Failed to create WS1 namespace 'ws1' 
YYYY-MM-DDT07:02:59.943Z [tomcat-exec-19 [] ERROR com.vmware.vcenter.trustmanagement.authbroker.BrokerClient  opId=] Failed to get a access token on host <vc_fqdn> for tenant HWS 
YYYY-MM-DDT07:02:59.943Z [tomcat-exec-19 [] ERROR com.vmware.vcenter.trustmanagement.migration.IdpReplacer  opId=] Replace operation failed. Attempting rollback. Triggering exception is: Failed to get a access token on host <vc_fqdn> for tenant HWS 
YYYY-MM-DDT07:02:59.943Z [tomcat-exec-19 [] ERROR com.vmware.vcenter.trustmanagement.migration.IdentityMigration  opId=] Error changing identity provider configuration: Failed to get a access token on host <vc_fqdn> for tenant HWS 

2. /var/log/vmware/vc-ws1a-broker/federation-service.log 
 
YYYY-MM-DDT06:57:15,027 WARN  <vc_fqdn>:federation (ForkJoinPool-2-worker-1) [-;-;-;-;-;-] com.vmware.vidm.common.gateway.mesh.GatewayAuthProvider - Failed to acquire token, returning cached token - Optional.empty, ScBadRequestException[Operation: POST -> http://localhost:10114/acs/token][Status:400]

 

3. /var/log/vmware/vc-ws1a-broker/token-service.log


YYYY-MM-DDT06:57:04,150 INFO  <vc_fqdn>:token (main) [;;;;] com.vmware.vidm.common.http.client.vertx.CircuitBreakerHttpClient - Created client with circuit breaker options io.vertx.circuitbreaker.CircuitBreakerOptions@3cc7cf58[maxFailures=1000,resetTimeout=30000,requestTimeout=-1] 
YYYY-MM-DDT06:57:04,140 WARN  <vc_fqdn>:token (vert.x-eventloop-thread-3) [;;;;] com.vmware.vidm.common.async.RetryCompletableFuture - Failed after max retries: 0 java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114 
YYYY-MM-DDT06:57:04,154 INFO  <vc_fqdn>:token (main) [;;;;] com.vmware.vidm.common.gateway.mesh.GatewayClientFactory - Initialized gateway [Class:class com.vmware.vidm.crypto.CryptoOperations][Endpoint:http://localhost:10111][Auth:true] 
YYYY-MM-DDT06:57:04,158 INFO  <vc_fqdn>:token (main) [;;;;] com.vmware.vidm.common.http.client.ClientConfig - Using system proxy configuration : Optional.empty 
YYYY-MM-DDT06:57:04,164 INFO  <vc_fqdn>:token (main) [;;;;] com.vmware.vidm.common.http.client.vertx.CircuitBreakerHttpClient - Created client with circuit breaker options io.vertx.circuitbreaker.CircuitBreakerOptions@61c8139a[maxFailures=1000,resetTimeout=30000,requestTimeout=-1] 
YYYY-MM-DDT06:57:04,155 WARN  <vc_fqdn>:token (ForkJoinPool-2-worker-3) [;;;;] com.vmware.vidm.common.resiliency.circuitbreaker.CircuitBreakers - Exception during execution inside circuit breaker LOCALHOST java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114 
YYYY-MM-DDT06:57:04,211 INFO  <vc_fqdn>:token (main) [;;;;] com.vmware.vidm.token.signing.TokenSigningServiceAutoConfiguration - Using crypto service to sign token 
YYYY-MM-DDT06:57:04,217 WARN  <vc_fqdn>:token (ForkJoinPool-2-worker-3) [;;;;] com.vmware.vidm.common.gateway.mesh.GatewayAuthProvider - Failed to acquire token, returning cached token - Optional.empty, io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114 

 

 

Environment

vCenter 8.x

Cause

vc-ws1a-broker service configuration files get lost which leaves the vc-ws1a-broker service in a broken state. The drift check failure is caused by the VMware Identity services being unavailable due to which it fails to get the access token.

Resolution

To resolve this issue, perform a manual secret rotation for the VMware Identity Services to restore the vc-ws1a-broker service configuration and enable successful access token retrieval using the script Rotate_WS1B_secrets.sh.

Note:

1. You can skip Step 6 if your vCenter Server is not configured with an external Identity Provider (IdP).

2. Before implementing the action plan it is mandatory to take snapshot of the vCenter VM. 

 

  1. Rotate WS1B secrets
    1. Download the Rotate_WS1B_secrets.sh script attached to this KB and upload to vCenter using WinSCP.
    2. SSH to vCenter with root user
    3. Change the script as executable
      chmod +x Rotate_WS1B_secrets.sh
    4. Fix CRLF to LF
      sed -i -e 's/\r//g' Rotate_WS1B_secrets.sh
    5. Run the script on the VC to rotate WS1 broker service secrets
      ./Rotate_WS1B_secrets.sh
  2. Get system token using the new operator_client_id and secret
    1. Get the operator client ID and secret from the masterapp credentials file
      cat /var/lib/ws1/masterapp-credentials.json
      Sample output:
      { "clientId": "operator_client_id", "clientSecret":"Bnv########################v4o=", "scopes":"admin system"}
    2. Call WS1B token endpoint to get a token with these credentials
      curl -k -u 'operator_client_id:<OPERATOR CLIENT SECRET>' 'https://<VC HOSTNAME>/acs/t/HWS/token' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials'
  3. Delete keys for tenants operator and customer
    • Using the token from step 3, call the following APIs to delete keys for customer and operator tenant
      • Delete keys for customer tenant
        curl -k --location --request DELETE 'https://<VC HOSTNAME>/crypto/t/CUSTOMER/jwt/keys' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'
      • Delete keys for operator tenant
        curl -k --location --request DELETE 'https://<VC HOSTNAME>/crypto/t/OPERATOR/jwt/keys' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'
      • To verify keys are deleted, run the following command
        psql -U postgres VCDB -c "SELECT * FROM vidm_schema.\"Crypto_Keys\" WHERE \"keyContainer\" in ('CUSTOMER:JWT', 'OPERATOR:JWT');"
        This should return zero rows.

  4. Generate new keys for tenants operator and customer
    • Using the token from step 3, call the following APIs to generate new keys for customer and operator tenant
      • Generate new keys for customer tenant
        curl -k --location --request POST 'https://<VC HOSTNAME>/crypto/t/CUSTOMER/jwt/keys?algorithmId=ALG_RSA_2048_SHA256' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'
      • Generate new keys for operator tenant
        curl -k --location --request POST 'https://<VC HOSTNAME>/crypto/t/OPERATOR/jwt/keys?algorithmId=ALG_RSA_2048_SHA256' \
--header 'Authorization: Bearer <SYSTEM TOKEN>'


  5. Update HVC Settings store with new master appcredentials.
    • Step 1 generates new master app credentials for WS1B service but the settings store still has the old credentials. To pick up the new credentials, follow these two steps
      1. Clear the credentials currently stored in settings store:
        1. First get a VC session
          curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '<Admin user>:<Admin password>'
        2. Use sessionID from above step to clear settings store
          curl -X PUT -k -H "vmware-api-session-id:<SESSION_ID>" http://localhost:1080/api/vcenter/settings/namespaces/ws1/items -d '{"items": {"master_client_secret": ""}}' -H "Content-Type: application/json"
      2. Restart Trustmanagement service
        vmon-cli -r trustmanagement
  6. Reset Client Secret in IDP object
    • Call the Authbroker Update API to reset the External IDP Client ID. Use the same client ID that was used to setup the IDP before upgrade or generate a new client ID and use it here.

      Note: Client secret can be obtained from the Okta server.
      Log into Okta server as Admin, search for the Oauth2 app created for the VC by clientID.
      We can recover the existing secret from under "CLIENT SECRETS" or generate a new one
      • First get a VC session
        curl -k --request POST --url https://localhost/rest/com/vmware/cis/session -u '<Admin user>:<Admin password>'
      • Call Authbroker API LIST to get IDP ID
        curl -k --location --request GET 'https://localhost/api/vcenter/identity/authbrokeridp' --header 'vmware-api-session-id: <VC SESSION ID>' | jq

        Note IPD ID of the IDP configured with customer tenant
      • Call Authbroker API Update to reset the client secret
        curl -k --location --request PATCH 'https://localhost/api/vcenter/identity/authbrokeridp/<IDP ID>' --header 'vmware-api-session-id: <VC SESSION ID>' --header 'Content-Type: application/json' --data-raw '{"client_secret" : "<CLIENT SECRET>"}'
  7. Restart Services
    1. Restart WS1B service
      vmon-cli -r vc-ws1a-broker
    2. Restart Trustmanagement service
      vmon-cli -r trustmanagement
    3. Restart STS service
      vmon-cli -r sts

After executing the above steps the Microsoft Entra Id should be configured successfully .

Attachments

Rotate_WS1B_secrets.sh get_app
Powered by Wolken Software