Unable to use SSO with VMware Cloud Foundation Operations: Login failed with "Failed to log in with VCF SSO service"

search cancel

Unable to use SSO with VMware Cloud Foundation Operations: Login failed with "Failed to log in with VCF SSO service"

book

Article ID: 429013

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

Users are unable to log in to VMware Cloud Foundation (VCF) Operations using Single Sign-On (SSO). The login attempt fails immediately with the error message: Failed to log in with VCF SSO service

An analysis of the /storage/log/vcops/log/analytics-*.log files reveals the following reveals a sequence of errors indicating a conflict during the OAuth client secret rotation, eventually leading to authentication failures.
Secret Rotation Conflict (HTTP 400): Initial logs show an attempt to rotate the secret failing because a rotation was already detected in progress.

ERROR analytics [ops@#### threadId="#####" threadName="DistTaskvIDB Secret Rotation Task"] [com.vmware.vrops.vidb.client.VidbClient.rotateSecret] - Failed to rotate secret of OAuth Client '<CLIENT_ID>'. Error: Request failed, returned error: HTTP/1.1 400 Bad Request, response body: {"errors":[{"code":"oauth2.client.secondary.secret.not.empty","message":"Client secret rotation already in progress. Please end rotation and retry.","parameters":{}}]}
com.vmware.vrops.vidb.client.exception.VidbClientException: Request failed, returned error: HTTP/1.1 400 Bad Request, response body: {"errors":[{"code":"oauth2.client.secondary.secret.not.empty","message":"Client secret rotation already in progress. Please end rotation and retry.","parameters":{}}]}
 at com.vmware.vrops.vidb.client.VidbClient.executeRequest(VidbClient.java:547) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vrops.vidb.client.VidbClient.rotateSecret(VidbClient.java:251) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vrops.vidb.client.VidbClient.rotateSecret(VidbClient.java:231) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.auth.vidb.task.VidbClientSecretRotationTask.run(VidbClientSecretRotationTask.java:54) ~[vcops-auth-server-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.platform.distributedtask.DistributedTaskExecutor$TaskProcessorThread.run(DistributedTaskExecutor.java:576) ~[alive_platform.jar:?]
 at com.integrien.alive.common.util.BaseThread$BaseThreadRunnable.run(BaseThread.java:177) ~[vrops-adapters-sdk.jar:?]
 at java.lang.Thread.run(Unknown Source) ~[?:?]
ERROR analytics [ops@#### threadId="#####" threadName="DistTaskvIDB Secret Rotation Task"] [com.vmware.vcops.auth.vidb.task.VidbClientSecretRotationTask.run] - Failed to rotate vIDB Client Secret

Authentication Failure (HTTP 401): After a while, the system begins failing to fetch tokens completely.

ERROR analytics [ops@#### threadId="#####" threadName="DistTaskvIDB Secret Rotation Task"] [com.vmware.vrops.vidb.client.VidbClient.getAccessTokenByClientCredentials] - Failed to fetch token by client credentials. Error: Request failed, returned error: HTTP/1.1 401 Unauthorized com.vmware.vrops.vidb.client.exception.VidbClientAuthException: Request failed, returned error: HTTP/1.1 401 Unauthorized
 at com.vmware.vrops.vidb.client.VidbClient.executeRequest(VidbClient.java:545) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vrops.vidb.client.VidbClient.getAccessTokenByClientCredentials(VidbClient.java:301) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.auth.vidb.task.VidbClientSecretRotationTask.run(VidbClientSecretRotationTask.java:42) ~[vcops-auth-server-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.platform.distributedtask.DistributedTaskExecutor$TaskProcessorThread.run(DistributedTaskExecutor.java:576) ~[alive_platform.jar:?]
 at com.integrien.alive.common.util.BaseThread$BaseThreadRunnable.run(BaseThread.java:177) ~[vrops-adapters-sdk.jar:?]
 at java.lang.Thread.run(Unknown Source) ~[?:?]

The following errors will also be seen in the /storage/log/vcops/log/analytics-*.log file:

[Date] [Time] ERROR analytics ID [ops@ID threadId="######" threadName="DistTaskvIDB Secret Rotation Task"] [com.vmware.vrops.vidb.client.VidbClient.getAccessTokenByClientCredentials] - Failed to fetch token by client credentials. Error: Request failed, returned error: HTTP/1.1 401 Unauthorized com.vmware.vrops.vidb.client.exception.VidbClientAuthException: Request failed, returned error: HTTP/1.1 401 Unauthorized
[Date] [Time] ERROR analytics ID [ops@ID threadId="######" threadName="DistTaskvIDB Secret Rotation Task"] [com.vmware.vcops.auth.vidb.task.VidbClientSecretRotationTask.run] - Failed to rotate vIDB Client Secret com.vmware.vrops.vidb.client.exception.VidbClientAuthException: Request failed, returned error: HTTP/1.1 401 Unauthorized

Environment

VCF Operations 9.0.x
VMware Identity Broker 9.0.x (External VIDB)

Cause

This issue originates from a network partition that creates a split-brain scenario between the Primary and Replica nodes. During this isolation, the original Primary node successfully rotates the internal VIDB client secret. However, when the partition heals and the cluster rejoins, the system prioritizes the database state of the promoted Replica node, which still holds the old, unrotated secret. This action effectively overwrites the valid new secret with stale data. Consequently, once the old secret expires, the application attempts to authenticate using invalid credentials, resulting in permanent 401 Unauthorized errors.

Resolution

Note: Reverting to a snapshot in VCF 9.x may disrupt integration with VCF SSO / VIDB.

VCF 9.0.x: This issue occurs frequently; snapshot reversions in this version carry a high probability of disrupting the integration.
VCF 9.1 and newer: The risk is significantly reduced, though there remains a slight probability of encountering the issue.

If the integration is broken, please refer to the steps below to restore connectivity.

Note: Emphasize to take a snapshot of the VCF Operations nodes.

Follow the below steps to rotate the secret:

SSH as root to the primary node of VCF Operations.

Phase 1: Authentication & Discovery

Step 1: Fetch VCF SSO Auth Source ID
Retrieve all authentication sources to identify the VIDB source ID. Endpoint is unauthenticated.

curl -X 'GET' \
  'https://<OPS_HOST>/suite-api/api/auth/sources' \
  -H 'accept: application/json'

Step 2: Acquire Operations Token
Obtain an authentication token for the admin user. This token is required for all subsequent steps.

curl -X 'POST' \
  'https://<OPS_HOST>/suite-api/api/auth/token/acquire' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "username": "<ADMIN_USERNAME>",
  "password": "<ADMIN_PASSWORD>"
}'

Note: Save the token value from the response. It will be used as <OPS_TOKEN> in headers below.

Step 3: Retrieve Current Auth Source Configuration
Fetch the existing configuration. Modify the following JSON structure for the final step.

curl -X 'GET' \
  'https://<OPS_HOST>/suite-api/api/auth/sources/<AUTH_SOURCE_ID>' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>'

Phase 2: Secret Rotation

Before patching the source, generate a new secret for the VCF Ops Client.

Step 4: Identify the VIDB Resource ID
Identify the internal ID of the VIDB based on the hostname.

curl -X 'GET' \
  'https://<OPS_HOST>/suite-api/internal/vidb/vidbs' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>' \
  -H 'x-vrealizeops-api-use-unsupported: true'

Incase if the above command fails with an error " {"type":"Error","message":"Query Parameter \"vcfId\" is required.","httpStatusCode":400,"apiErrorCode":1509}" , you need to include the vcfId as a query parameter, as shown below:

curl -X 'GET' \
  'https://<OPS_HOST>/suite-api/internal/vidb/vidbs?vcfId=<vcf-instance-id>' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>' \
  -H 'x-vrealizeops-api-use-unsupported: true'

Note: <vcf-instance-id> can be found from the adapters API. Here are the steps:

Login to VCF Operations UI with admin privileges
Go to </> Developer Center > APIs & SDKs > Operations API, then launch the "API documentation"
Authorize the page using "admin" or equivalent permissions
Navigate to Adapters > GET /api/adapters > Click on "Try it out", then select Execute
In the response body output, check for "VMware Cloud Foundation Adapter" and note down the "id"
Use this id as the <vcf-instance-id> in the above query parameter

Step 5: Rotate Secret for the VCF Ops Client (If you wish to retain existing user and group role assignments for VCF SSO users, please follow the steps outlined below. Alternatively, if reassigning permissions is acceptable, you can resolve the issue by unjoining VCF Operations from VCF SSO and then rejoining it.)
Execute the rotation.

Target Client ID: Follow these steps to identify the correct clientId for your VCF Operations environment:

SSH into the VCF Operations Node as the root user.

Connect to the database by executing the following command:

su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -p 5433 -d vcopsdb"

Identify the AuthSource record for VCF Operations by running the query below:
```
select * from kv_vidb_auth_sources where vcf_component_type='VCF_OPS';
```

Use the value of clientId from the above output.

Resource ID: Use the ID found in Phase 2 Step 4.

curl --request PUT \
  'https://<OPS_HOST>/suite-api/internal/vidb/oauthclients/<CLIENT_ID>/secrets/rotate?vidbResourceId=<RESOURCE_ID>' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>' \
  -H 'x-vrealizeops-api-use-unsupported: true'

Critical: The API response will return the new client secret. Copy this value immediately for use in the final step.

Phase 3: Apply Configuration Patch

Step 6: Update Auth Source with New Credentials
Modify the configuration retrieved in Step 3 and apply the changes.

curl -X 'PUT' \
  'https://<OPS_HOST>/suite-api/api/auth/sources' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{
  "id": "<AUTH_SOURCE_ID>",
  "name": "VCF SSO",
  "sourceType": {
    "id": "VIDB",
    "name": "VIDB"
  },
  "created": 0,
  "lastModified": 0,
  "property": [
    {
      "name": "display-name",
      "value": "VCF SSO"
    },
    {
      "name": "issuer-url",
      "value": "https://<VCF_SSO_HOST>/acs/t/<TENANT_NAME>"
    },
    {
      "name": "client-id",
      "value": "<CLIENT_ID>"
    },
    {
      "name": "client-secret",
      "value": "<NEW_CLIENT_SECRET>"
    }
  ],
  "certificates": []
}'

Additional Information

Placeholder	Description
<OPS_HOST>	VCF Operations Manager hostname or IP
<ADMIN_USERNAME>	VCF Ops admin username
<ADMIN_PASSWORD>	VCF Ops admin password
<OPS_TOKEN>	Operations token from Step 2
<AUTH_SOURCE_ID>	Authentication source UUID from Step 1
<VCF_SSO_HOST>	VCF SSO server hostname
<TENANT_NAME>	SSO tenant name
<CLIENT_ID>	OAuth2 client ID
<CLIENT_SECRET>	OAuth2 client secret

Feedback

thumb_up Yes

thumb_down No