vCenter Login Failure with Error "[400] Unable to authenticate. Check your credentials. If problem persists, contact your administrator." in VCF 9.x
search cancel

vCenter Login Failure with Error "[400] Unable to authenticate. Check your credentials. If problem persists, contact your administrator." in VCF 9.x

book

Article ID: 414489

calendar_today

Updated On:

Products

VMware vCenter Server VMware Cloud Foundation VCF Operations

Issue/Introduction

  • Login with "VCF SSO" fails for Management domain and Workload domain VCs with the following error:

[400] Unable to authenticate. Check your credentials. If problem persists, contact your administrator.

 

  • vSphere client logs show error "oauth2.authorization.credentials.invalid" :

/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

YYYY-MM-DDTHH:MIN:SEC [ERROR] p-nio-127.0.0.1-5090-exec-12 70000572 100326 ###### com.vmware.skyscraper.oauth2.common.Oau
th2Helper                  login redirect.  Discovery page https://<VCF_IDB_or_Mgmt_VC_with_embedded_IDB_FQDN>/acs/t/CUSTOMER/authorize?clien
t_id=#####-####-####-####-########&redirect_uri=https://<Affected_VC_FQDN>/ui/login/oauth2/authcode&state
=#####-####-####-####-########&response_type=code, state ##-##-##-####
YYYY-MM-DDTHH:MIN:SEC [ERROR] p-nio-127.0.0.1-5090-exec-12 70000572 100326  ###### c.v.vsphere.client.security.oauth2.Oaut
h2CodeResponseHandler      Received an Oauth2 Authorization code. Processing it now for Auth Token and exchange...
...
YYYY-MM-DDTHH:MIN:SEC [ERROR] p-nio-127.0.0.1-5090-exec-12 70000572 100326 ###### com.vmware.skyscraper.oauth2.common.Oauth2Helper Exception while exchanging token with csp with for code ###### and state ##-##-##-####. Csp responded with status 401 UNAUTHORIZED  and body "error":"invalid_client","error_description":"oauth2.authorization.credentials.invalid"}
YYYY-MM-DDTHH:MIN:SEC [ERROR] p-nio-127.0.0.1-5090-exec-12 70000572 100326 ###### c.v.vsphere.client.security.oauth2.Oauth2CodeResponseHandler Oauth2 Authorization code assertion failed java.lang.RuntimeException: Generating Authorization Token has an exception
        at com.vmware.skyscraper.oauth2.common.Oauth2Helper.handleAuthTokenRequest(Oauth2Helper.java:201)
        at com.vmware.skyscraper.oauth2.common.Oauth2Helper.handleAuthTokenRequest(Oauth2Helper.java:117)

  • vIDB logs on the affected vCenter Server show "horizon.credentials.invalid" 

  • In case of external vIDB, Check pod logs for the vidb-service using the below steps:
    • Connect to the vIDB nodes and login as vmware-system-user and switch to root user using the su command
    • Execute the following command to enable the kubectl commands: export KUBECONFIG=/etc/kubernetes/admin.conf
    • Execute the following command to list all the pods: kubectl get pods -A | grep vidb-service
    • Check pod logs for the vidb-service using the command: kubectl logs -n vidb-external   vidb-service-8#######5 | less
    • Sample log entry in case of external vIDB:
      YYYY-MM-DDTHH:MIN:SEC WARN  vidb-service-8#######5:accesscontrol (ForkJoinPool-5-worker-3) [CUSTOMER;-;##.##.##.##;ab############fa;-;fc###################91;client_credentials] com.vmware.vidm.accesscontrol.resource.auth.TokenResource - Failed during issuing token java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: com.vmware.vidm.accesscontrol.exceptions.oauth2.UnauthorizedClientException: oauth2.authorization.credentials.invalid

  • /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log

    YYYY-MM-DDTHH:MIN:SEC WARN  <Affected_VC_FQDN> (ForkJoinPool-5-worker-4) [CUSTOMER;-;127.0.0.1;####-##-####; ####-##-####;authorization_code] com.vmware.vidm.accesscontrol.OAuth2ClientService - [getValidatedClient] Unable to verify client secret.
    YYYY-MM-DDTHH:MIN:SEC WARN  <Affected_VC_FQDN> (ForkJoinPool-5-worker-4) [CUSTOMER;-;127.0.0.1;##-##-##; ##-##-##;authorization_code] com.vmware.vidm.accesscontrol.resource.auth.TokenResource - Failed during issuing token java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: com.vmware.vidm.accesscontrol.exceptions.oauth2.UnauthorizedClientException: oauth2.aut
    horization.credentials.invalid
            at com.vmware.vidm.accesscontrol.resource.auth.TokenResource.lambda$getToken$3(TokenResource.java:351)
            at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)

Environment

  • VMware Cloud Foundation 9.0
  • vCenter Server 9.0.x

Cause

This can happen when the client credentials used by the vCenter's oauth2 client becomes invalid because it has expired. In VCF 9.0, there is a logic to rotate the VC's OAuth2 client credentials every 90 days. A task is triggered 90 days after the client is first created (i.e. at the time VCF SSO is configured). After rotating the secret, the task tries to update the secret in VC's provider info stored in Trustmanagement service (TMS) but update operation fails. This leaves the VC in a state where the VC's OAuth2 client has new credentials, but it is not updated in TMS. During login operation, the UI fetches VC's provider info from TMS which has the outdated client credentials and the login workflow fails.

This can be further validated from TMS logs which reports "Failed to update client secret" as shown below:

/var/log/vmware/trustmanagement/trustmanagement-svcs.log

YYYY-MM-DDTHH:MIN:SEC [pool-12-thread-1 [] ERROR com.vmware.iam.txaz.secretrotation.ClientSecretRotator  opId=] Rotation the secret of client ####-##-##failed
com.vmware.iam.txaz.exception.TxazException: Failed to update client secret in TMS for Identity Provider: CUSTOMER
        at com.vmware.vcenter.trustmanagement.authbroker.SsoOAuthAppSecretExpiryChecker$VcClientCredentialProvider.setClientSecret(SsoOAuthAppSecretExpiryChecker.java:188) ~[libservice.jar:?]
        at com.vmware.iam.txaz.secretrotation.ClientSecretRotator.rotateSecret(ClientSecretRotator.java:118) ~[vc-txaz-client.jar:?]
        at com.vmware.iam.txaz.secretrotation.ClientSecretRotator.lambda$initSecretRotationTask$0(ClientSecretRotator.java:71) ~[vc-txaz-client.jar:?]
        at com.vmware.iam.txaz.secretrotation.ClientSecretRotator.lambda$errorHandlingWrapper$1(ClientSecretRotator.java:85) ~[vc-txaz-client.jar:?]
Caused by: com.vmware.vcenter.trustmanagement.impl.VCFBrokerRestrictionException: VC restricts update/create API call for VMWARE_SSO_FEDERATION, Please delete the IDP and create another one
        at com.vmware.vcenter.trustmanagement.impl.VcIdentityProviders.update(VcIdentityProviders.java:389) ~[libservice.jar:?]
        at com.vmware.vcenter.trustmanagement.impl.VcIdentityProviders.update(VcIdentityProviders.java:369) ~[libservice.jar:?]

Resolution

This issue is resolved in vCenter 9.0.2. Log in to the Broadcom Support Portal to download this patch, depending on your entitlement, VMware vSphere Foundation or VMware Cloud Foundation.

Note: If you are experiencing this issue on 9.0.0 or 9.0.1, apply the workaround below to resolve the issue. Afterwards, upgrade vCenter to 9.0.2 to prevent the issue from recurring.

Workaround:

Recreate the IDP configuration for VC or update VC's IDP configuration with the new client secret.

  1. Workload Domain vCenters and Management Domain vCenter with External IDB
  2. Management Domain vCenter with Embedded IDB (follow any of below method)

Mandatory Steps

  1. Before proceeding with below mentioned steps, take a backup of the affected vCenter server.
  2. Highlighted entries in resolution steps, to be replaced with actual values.

Workload Domain vCenters and Management Domain vCenter with External Identity Broker

Reregister the VCF SSO Configuration

For the workload domain VCs, follow below mentioned steps from the UI to re-register the SSO configuration. 

  1. Login to VCF Operations > Identity & Access > VCF Instances > Select the instance > Component Configuration
  2. Select the Workload Domain VC and click on Deregister Component
  3. Select again the Workload Domain VC and click on Configure Component
  4. Follow on-screen instructions to complete the configuration.

 

Management Domain vCenter with Embedded Identity Broker

For management domain VCs with embedded IDB, there is no way to recreate the authsource in the UI, so this must be done manually using one of the methods below.

Method 1: This involves manually creating a new IDP configuration for VC.

Log in to vCenter as.'root' user via SSH, then follow the steps below:

  1. Get VC Session

    curl -k --request POST --url https://<MGMT VC FQDN>/rest/com/vmware/cis/session -u '<Username>:<password>'

    Note: Username is [email protected]

  2. Get Tenant Admin Client Token

    curl -k --location --request GET 'https://<MGMT VC FQDN>/api/vcenter/identity/broker/tenants/CUSTOMER/admin-client' --header 'vmware-api-session-id: <session from step 1>' | jq

  3. Create a new OAuth client for the MGMT domain VC.  The client_id and secret fields are randomly generated UUIDs (it can be any random string). [Create a random UUID for client_id & secret]

    curl -k --request POST \
      --url https://<MGMT VC FQDN>/acs/t/CUSTOMER/broker/oauth2-clients \
      --header 'authorization: Bearer <tenant admin client token from step 2>' \
      --header 'content-type: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.with.rule.sets+json' \
      --data '{
        "access_token_ttl": 30,
        "client_id": "4######a-####-####-####-9#########21",
        "secret":"0######b-####-####-####-7#########27",
        "grant_types": [
            "authorization_code",
            "client_credentials",
            "refresh_token",
            "password"
        ],
        "refresh_token_idle_ttl": 1440,
        "refresh_token_ttl": 1440,
        "rotate_secret": true,
        "redirect_uris": [
            "https://<MGMT VC FQDN>/ui/login/oauth2/authcode"
        ],
        "rule_set_names": [
            "READ_ONLY_TENANT_ADMIN"
        ],
        "scope": [
            "admin",
            "user",
            "openid",
            "profile",
            "group"
        ]
    }'

  4. Note down VC's provider info by doing a GET on the configuration: 

    curl -k --request GET \
      --url https://<MGMT VC FQDN>/api/vcenter/identity/providers/CUSTOMER \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session from step 1>'

  5. Delete VC's provider configuration. 

    curl -k --request DELETE \
      --url https://<MGMT VC FQDN>/rest/vcenter/identity/providers/CUSTOMER \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session from step 1>'

  6. Create a new provider for the MGMT domain VC using the client created in step 3. The client_id and client_secret fields are the values used to create the OAuth client in step 3.

    curl -k --request POST \
      --url https://<MGMT VC FQDN>/api/vcenter/identity/providers \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session ID from step 1' \
      --data '{
      "federation_type": "VMWARE_SSO_FEDERATION",
      "provider": "CUSTOMER",
      "name": "VCF SSO",
      "config_tag": "Oidc",
      "is_default": true,
      "oidc": {
        "discovery_endpoint": "https://<MGMT VC FQDN>:443/acs/t/CUSTOMER/.well-known/openid-configuration",
        "client_id": "4######a-####-####-####-9#########21",
        "client_secret": "0######b-####-####-####-7#########27",
        "claim_map": {}
      },
      "idm_protocol": "SCIM2_0",
      "idm_endpoints": [
        "https://<MGMT VC FQDN>/usergroup/t/CUSTOMER/scim/v2/Groups",
        "https://<MGMT VC FQDN>/usergroup/t/CUSTOMER/scim/v2/Users"
      ],
      "auth_query_params": {
        "response_type": [
          "code"
        ]
      },
      "upn_claim": "acct",
      "groups_claim": "group_names",
      "domain_names": ["<domains from the existing provider here, which noted down in step 4>"]
    }'

  7. Edit the Trust Management service configuration file (trustmanagement-vlsi.xml) to remove the ssoAppSecretExpiryChecker scheduled task to avoid the same issue in the future. Execute the 'sed' command mentioned below:

    sed -i.bak \
    -e '/<bean id="ssoAppSecretExpiryChecker"/,/<\/bean>/d' \
    -e '/<task:scheduler id="ssoAppSecretExpiryCheckerScheduler"\/>/d' \
    -e '//d' \
    -e '/ <!-- Check for SSO OAuth App Client Secret Expiry in VIDB by instantiating TxazClient -->/d' \
    -e '/<task:scheduled-tasks scheduler="ssoAppSecretExpiryCheckerScheduler"/,/<\/task:scheduled-tasks>/d' \
    /usr/lib/vmware-trustmanagement/config/trustmanagement-vlsi.xml

  8. Restart the Trust Management service.

    service-control --restart vmware-trustmanagement

Method 2: Manually rotate the VC's OAuth2 client secret and then patch the IDP configuration with the new secret.

Log in to vCenter as 'root' user via SSH, then follow the steps below:

  1. Get VC Session

    curl -k --request POST --url https://<MGMT VC FQDN>/rest/com/vmware/cis/session -u '<Username>:<password>'

    Note: Username is [email protected]

  2. Get Tenant Admin Client Token

    curl -k --location --request GET 'https://<MGMT VC FQDN>/api/vcenter/identity/broker/tenants/CUSTOMER/admin-client' --header 'vmware-api-session-id: <session id from step 1>' | jq

  3. Get VC's provider info and note down the "client_id" field from the output below 

    curl -k --request GET \
      --url https://<MGMT VC FQDN>/api/vcenter/identity/providers/CUSTOMER \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session from step 1>'
  4. Rotate the secret for this client. This is a two-step process.

    1. Initiate secret rotation, use client_id from above, and use a randomly generated string as secret

      curl -k --request POST \
        --url 'https://<MGMT VC FQDN>/acs/t/CUSTOMER/broker/oauth2-clients/<CLIENT ID>?action=start-rotate-secret' \
        --header 'accept: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.secret.rotation+json' \
        --header 'Content-Type: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.secret.rotation+json' \
        --header 'authorization: Bearer <tenant admin client token here from step 2>' \
        --data '{
        "primary_secret_auto_retire_duration": 1,
        "secondary_secret": "<new random secret>"
      }'

    2. End secret rotation. This leads to OAuth2 client secret rotation.

      curl -v -k --request POST \
        --url 'https://<MGMT VC FQDN>/acs/t/CUSTOMER/broker/oauth2-clients/<CLIENT ID>?action=retire-primary-secret' \
        --header 'accept: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.secret.rotation+json' \
        --header 'Content-Type: application/vnd.vmware.horizon.manager.accesscontrol.broker.oauth2client.secret.rotation+json' \
        --header 'authorization: Bearer <tenant admin client token from step 2 >' \
        --data '{}'

  5. Update VC's IDP configuration in TMS 

    curl -k --request PATCH \
      --url https://<MGMT VC FQDN>/api/vcenter/identity/providers/CUSTOMER \
      --header 'content-type: application/json' \
      --header 'vmware-api-session-id: <session from step 1>' \
      --data '{
      "config_tag": "Oidc",
      "federation_type": "VMWARE_SSO_FEDERATION",
      "oidc": {
        "client_secret": "<secret id from step 4>"
      }
    }'

  6. Edit the Trust Management service configuration file (trustmanagement-vlsi.xml) to remove the ssoAppSecretExpiryChecker scheduled task to avoid the same issue in the future. Execute the 'sed' command mentioned below:

    sed -i.bak \
    -e '/<bean id="ssoAppSecretExpiryChecker"/,/<\/bean>/d' \
    -e '/<task:scheduler id="ssoAppSecretExpiryCheckerScheduler"\/>/d' \
    -e '//d' \
    -e '/ <!-- Check for SSO OAuth App Client Secret Expiry in VIDB by instantiating TxazClient -->/d' \
    -e '/<task:scheduled-tasks scheduler="ssoAppSecretExpiryCheckerScheduler"/,/<\/task:scheduled-tasks>/d' \
    /usr/lib/vmware-trustmanagement/config/trustmanagement-vlsi.xml

  7. Restart the Trust Management service.

    service-control --restart vmware-trustmanagement
Powered by Wolken Software