"[400] Unable to authenticate. Check your credentials", login failure on 9.0 VCs due to invalid OAuth2 client
search cancel

"[400] Unable to authenticate. Check your credentials", login failure on 9.0 VCs due to invalid OAuth2 client

book

Article ID: 414489

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Login with "VCF SSO" fails for Management domain and Workload domain VCs with the following error:

[400] Unable to authenticate. Check your credentials. If problem persists, contact your administrator.

 

  • vSphere client logs shows error "oauth2.authorization.credentials.invalid"

/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

YYYY-MM-DDTHH:MIN:SEC [ERROR] p-nio-127.0.0.1-5090-exec-12 70000572 100326 ###### com.vmware.skyscraper.oauth2.common.Oauth2Helper Exception while exchanging token with csp with for code ###### and state ##-##-##-####. Csp responded with status 401 UNAUTHORIZED  and body "error":"invalid_client","error_description":"oauth2.authorization.credentials.invalid"}
YYYY-MM-DDTHH:MIN:SEC [ERROR] p-nio-127.0.0.1-5090-exec-12 70000572 100326 ###### c.v.vsphere.client.security.oauth2.Oauth2CodeResponseHandler Oauth2 Authorization code assertion failed java.lang.RuntimeException: Generating Authorization Token has an exception
        at com.vmware.skyscraper.oauth2.common.Oauth2Helper.handleAuthTokenRequest(Oauth2Helper.java:201)
        at com.vmware.skyscraper.oauth2.common.Oauth2Helper.handleAuthTokenRequest(Oauth2Helper.java:117)

  • VIDB logs shows error "credentials.invalid"

/var/log/vmware/vc-ws1a-broker/accesscontrol-service.log

YYYY-MM-DDTHH:MIN:SEC WARN  example.vcenter.local:accesscontrol (ForkJoinPool-5-worker-4) [CUSTOMER;-;127.0.0.1;####-##-####; ####-##-####;authorization_code] com.vmware.vidm.accesscontrol.OAuth2ClientService - [getValidatedClient] Unable to verify client secret.
YYYY-MM-DDTHH:MIN:SEC WARN  example.vcenter.local:accesscontrol (ForkJoinPool-5-worker-4) [CUSTOMER;-;127.0.0.1;##-##-##; ##-##-##;authorization_code] com.vmware.vidm.accesscontrol.resource.auth.TokenResource - Failed during issuing token java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: com.vmware.vidm.accesscontrol.exceptions.oauth2.UnauthorizedClientException: oauth2.aut
horization.credentials.invalid
        at com.vmware.vidm.accesscontrol.resource.auth.TokenResource.lambda$getToken$3(TokenResource.java:351)
        at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.postFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)

Environment

  • vCenter 9.0.0
  • vCenter 9.0.1

Cause

  • This can happen when the client credentials used by the vCenter's oauth2 client becomes invalid because it has expired. In VCF 9.0, there is a logic to rotate the VC's OAuth2 client credentials every 90 days. A task is triggered 90 days after the client is first created (i.e. at the time VCF SSO is configured). After rotating the secret, the task tries to update the secret in VC's provider info stored in Trustmanagement service (TMS) but update operation fails. This leaves the VC in a state where the VC's OAuth2 client has new credentials, but it is not updated in TMS. During login operation, the UI fetches VC's provider info from TMS which has the outdated client credentials and the login workflow fails.

  • TMS logs reports "Failed to update client secret"

/var/log/vmware/trustmanagement/trustmanagement-svcs.log

YYYY-MM-DDTHH:MIN:SEC [pool-12-thread-1 [] ERROR com.vmware.iam.txaz.secretrotation.ClientSecretRotator  opId=] Rotation the secret of client ####-##-##failed
com.vmware.iam.txaz.exception.TxazException: Failed to update client secret in TMS for Identity Provider: CUSTOMER
        at com.vmware.vcenter.trustmanagement.authbroker.SsoOAuthAppSecretExpiryChecker$VcClientCredentialProvider.setClientSecret(SsoOAuthAppSecretExpiryChecker.java:188) ~[libservice.jar:?]
        at com.vmware.iam.txaz.secretrotation.ClientSecretRotator.rotateSecret(ClientSecretRotator.java:118) ~[vc-txaz-client.jar:?]
        at com.vmware.iam.txaz.secretrotation.ClientSecretRotator.lambda$initSecretRotationTask$0(ClientSecretRotator.java:71) ~[vc-txaz-client.jar:?]
        at com.vmware.iam.txaz.secretrotation.ClientSecretRotator.lambda$errorHandlingWrapper$1(ClientSecretRotator.java:85) ~[vc-txaz-client.jar:?]
Caused by: com.vmware.vcenter.trustmanagement.impl.VCFBrokerRestrictionException: VC restricts update/create API call for VMWARE_SSO_FEDERATION, Please delete the IDP and create another one
        at com.vmware.vcenter.trustmanagement.impl.VcIdentityProviders.update(VcIdentityProviders.java:389) ~[libservice.jar:?]
        at com.vmware.vcenter.trustmanagement.impl.VcIdentityProviders.update(VcIdentityProviders.java:369) ~[libservice.jar:?]

Resolution

This issue is resolved in vCenter 9.0.2. Log in to the Broadcom Support Portal to download this patch, depending on your entitlement, VMware vSphere Foundation or VMware Cloud Foundation.

Note: If you are experiencing this issue on 9.0.0 or 9.0.1, apply the workaround below to resolve the issue. Afterward, upgrade vCenter to 9.0.2 to prevent the issue from recurring.

Workaround:

Recreate the IDP configuration for VC or update VC's IDP configuration with the new client secret.

Mandatory Steps

  • Before proceeding with below mentioned steps, please take a backup of vCenter server.
  • Highlighted entries in resolution steps, to be replaced with actual values.

Workload Domain VCs

Reregister the VCF SSO Configuration

For the workload domain VCs, follow below mentioned steps from the UI to re-register the SSO configuration. 

  1. Login to VCF Operations > Identity & Access > VCF Instances > Select the instance > Component Configuration
  2. Select the Workload Domain VC and click on Deregister Component
  3. Select again the Workload Domain VC and click on Configure Component
  4. Follow on-screen instructions to complete the configuration.

 

Management Domain VC

For the management domain VCs, there is no way to recreate the authsource from UI, so this has to be done manually by following either of the below mentioned methods.

Method 1: This involves manually creating a new IDP configuration for VC.

Login to vCenter as 'root' user via SSH, then follow steps below:

  1. Get VC Session

    curl -k --request POST --url https://<MGMT VC FQDN>/rest/com/vmware/cis/session -u '<Username>:<password>'

  2. Get Tenant Admin Client Token

    curl -k --location --request GET 'https://<MGMT VC FQDN>/api/vcenter/identity/broker/tenants/CUSTOMER/admin-client' --header 'vmware-api-session-id: <session from step 1>' | jq

  3. Create new OAuth client for MGMT domain VC.  The client_id and secret fields are randomly generated UUIDs (it can be any random string)

    curl -k --request POST \
      --url https://<MGMT VC FQDN>/acs/t/CUSTOMER/broker/oauth2-clients \
      --header 'authorization: Bearer <tenant admin client token from step 2>' \
      --header 'content-type: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.with.rule.sets+json' \
      --data '{
        "access_token_ttl": 30,
        "client_id": "4######a-####-####-####-9#########21",
        "secret":"0######b-####-####-####-7#########27",
        "grant_types": [
            "authorization_code",
            "client_credentials",
            "refresh_token",
            "password"
        ],
        "refresh_token_idle_ttl": 1440,
        "refresh_token_ttl": 1440,
        "rotate_secret": true,
        "redirect_uris": [
            "https://<MGMT VC FQDN>/ui/login/oauth2/authcode"
        ],
        "rule_set_names": [
            "READ_ONLY_TENANT_ADMIN"
        ],
        "scope": [
            "admin",
            "user",
            "openid",
            "profile",
            "group"
        ]
    }'

  4. Note down VC's provider info by doing a GET on the configuration: 

    curl -k --request GET \
      --url https://<MGMT VC FQDN>/api/vcenter/identity/providers/CUSTOMER \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session from step 1>'

  5. Delete VC's provider configuration. 

    curl -k --request DELETE \
      --url https://<MGMT VC FQDN>/rest/vcenter/identity/providers/CUSTOMER \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session from step 1>'

  6. Create new provider for MGMT domain VC using the client created in step 3. The client_id and client_secret fields are the values used to create the OAuth client in step 3.

    curl -k --request POST \
      --url https://<MGMT VC FQDN>/api/vcenter/identity/providers \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session ID from step 1' \
      --data '{
      "federation_type": "VMWARE_SSO_FEDERATION",
      "provider": "CUSTOMER",
      "name": "VCF SSO",
      "config_tag": "Oidc",
      "is_default": true,
      "oidc": {
        "discovery_endpoint": "https://<MGMT VC FQDN>:443/acs/t/CUSTOMER/.well-known/openid-configuration",
        "client_id": "4######a-####-####-####-9#########21",
        "client_secret": "0######b-####-####-####-7#########27",
        "claim_map": {}
      },
      "idm_protocol": "SCIM2_0",
      "idm_endpoints": [
        "https://<MGMT VC FQDN>/usergroup/t/CUSTOMER/scim/v2/Groups",
        "https://<MGMT VC FQDN>/usergroup/t/CUSTOMER/scim/v2/Users"
      ],
      "auth_query_params": {
        "response_type": [
          "code"
        ]
      },
      "upn_claim": "acct",
      "groups_claim": "group_names",
      "domain_names": ["<domains from the existing provider here, which noted down in step 4>"]
    }'

  7. Edit the Trust Management service configuration file (trustmanagement-vlsi.xml) to remove the ssoAppSecretExpiryChecker scheduled task to avoid the same issue in future. Execute the 'sed' command mentioned below:

    sed -i.bak \
    -e '/<bean id="ssoAppSecretExpiryChecker"/,/<\/bean>/d' \
    -e '/<task:scheduler id="ssoAppSecretExpiryCheckerScheduler"\/>/d' \
    -e '//d' \
    -e '/ <!-- Check for SSO OAuth App Client Secret Expiry in VIDB by instantiating TxazClient -->/d' \
    -e '/<task:scheduled-tasks scheduler="ssoAppSecretExpiryCheckerScheduler"/,/<\/task:scheduled-tasks>/d' \
    /usr/lib/vmware-trustmanagement/config/trustmanagement-vlsi.xml

  8. Restart the Trust Management service.

    service-control --restart vmware-trustmanagement

Method 2: Manually rotate the VC's OAuth2 client secret and then patch the IDP configuration with the new secret.

Login to vCenter as 'root' user via SSH, then follow steps below:

  1. Get VC Session

    curl -k --request POST --url https://<MGMT VC FQDN>/rest/com/vmware/cis/session -u '<Username>:<password>'

  2. Get Tenant Admin Client Token

    curl -k --location --request GET 'https://<MGMT VC FQDN>/api/vcenter/identity/broker/tenants/CUSTOMER/admin-client' --header 'vmware-api-session-id: <session id from step 1>' | jq

  3. Get VC's provider info and note down the "client_id" field from the below output 

    curl -k --request GET \
      --url https://<MGMT VC FQDN>/api/vcenter/identity/providers/CUSTOMER \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session from step 1>'
  4. Rotate the secret for this client. This is a two step process.

    1. Initiate secret rotation, use client_id from above and use a randomly generated string as secret

      curl -k --request POST \
        --url 'https://<MGMT VC FQDN>/acs/t/CUSTOMER/broker/oauth2-clients/<CLIENT ID>?action=start-rotate-secret' \
        --header 'accept: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.secret.rotation+json' \
        --header 'Content-Type: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.secret.rotation+json' \
        --header 'authorization: Bearer <tenant admin client token here from step 2>' \
        --data '{
        "primary_secret_auto_retire_duration": 1,
        "secondary_secret": "<new random secret>"
      }'

    2. End secret rotation. This leads to OAuth2 client secret rotation.

      curl -v -k --request POST \
        --url 'https://<MGMT VC FQDN>/acs/t/CUSTOMER/broker/oauth2-clients/<CLIENT ID>?action=retire-primary-secret' \
        --header 'accept: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.secret.rotation+json' \
        --header 'Content-Type: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.secret.rotation+json' \
        --header 'authorization: Bearer <tenant admin client token from step 2 >' \
        --data '{}'

  5. Update VC's IDP configuration in TMS 

    curl -k --request PATCH \
      --url https://<MGMT VC FQDN>/api/vcenter/identity/providers/CUSTOMER \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session from step 1>' \
      --data '{
      "config_tag": "Oidc",
      "federation_type": "VMWARE_SSO_FEDERATION",
      "oidc": {
        "client_secret": "<secret id from step 4>"
      }
    }'

  6. Edit the Trust Management service configuration file (trustmanagement-vlsi.xml) to remove the ssoAppSecretExpiryChecker scheduled task to avoid the same issue in future. Execute the 'sed' command mentioned below:

    sed -i.bak \
    -e '/<bean id="ssoAppSecretExpiryChecker"/,/<\/bean>/d' \
    -e '/<task:scheduler id="ssoAppSecretExpiryCheckerScheduler"\/>/d' \
    -e '//d' \
    -e '/ <!-- Check for SSO OAuth App Client Secret Expiry in VIDB by instantiating TxazClient -->/d' \
    -e '/<task:scheduled-tasks scheduler="ssoAppSecretExpiryCheckerScheduler"/,/<\/task:scheduled-tasks>/d' \
    /usr/lib/vmware-trustmanagement/config/trustmanagement-vlsi.xml

  7. Restart the Trust Management service.

    service-control --restart vmware-trustmanagement
Powered by Wolken Software