Unable to use SSO with VMware Cloud Foundation Operations: Login failed with "Failed to log in with VCF SSO service"

Products

VCF Operations

Issue/Introduction

Users are unable to log in to VMware Cloud Foundation (VCF) Operations using Single Sign-On (SSO). The login attempt fails immediately with the error message: Failed to log in with VCF SSO service

An analysis of the /storage/log/vcops/log/analytics-*.log files reveals the following reveals a sequence of errors indicating a conflict during the OAuth client secret rotation, eventually leading to authentication failures.
Secret Rotation Conflict (HTTP 400): Initial logs show an attempt to rotate the secret failing because a rotation was already detected in progress.

ERROR analytics 5810 [ops@4413 threadId="31846" threadName="DistTaskvIDB Secret Rotation Task"] [com.vmware.vrops.vidb.client.VidbClient.rotateSecret] - Failed to rotate secret of OAuth Client '<CLIENT_ID>'. Error: Request failed, returned error: HTTP/1.1 400 Bad Request, response body: {"errors":[{"code":"oauth2.client.secondary.secret.not.empty","message":"Client secret rotation already in progress. Please end rotation and retry.","parameters":{}}]}
com.vmware.vrops.vidb.client.exception.VidbClientException: Request failed, returned error: HTTP/1.1 400 Bad Request, response body: {"errors":[{"code":"oauth2.client.secondary.secret.not.empty","message":"Client secret rotation already in progress. Please end rotation and retry.","parameters":{}}]}
 at com.vmware.vrops.vidb.client.VidbClient.executeRequest(VidbClient.java:547) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vrops.vidb.client.VidbClient.rotateSecret(VidbClient.java:251) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vrops.vidb.client.VidbClient.rotateSecret(VidbClient.java:231) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.auth.vidb.task.VidbClientSecretRotationTask.run(VidbClientSecretRotationTask.java:54) ~[vcops-auth-server-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.platform.distributedtask.DistributedTaskExecutor$TaskProcessorThread.run(DistributedTaskExecutor.java:576) ~[alive_platform.jar:?]
 at com.integrien.alive.common.util.BaseThread$BaseThreadRunnable.run(BaseThread.java:177) ~[vrops-adapters-sdk.jar:?]
 at java.lang.Thread.run(Unknown Source) ~[?:?]
ERROR analytics 5810 [ops@4413 threadId="31846" threadName="DistTaskvIDB Secret Rotation Task"] [com.vmware.vcops.auth.vidb.task.VidbClientSecretRotationTask.run] - Failed to rotate vIDB Client Secret

Authentication Failure (HTTP 401): After a while, the system begins failing to fetch tokens completely.

ERROR analytics 5810 [ops@4413 threadId="32619" threadName="DistTaskvIDB Secret Rotation Task"] [com.vmware.vrops.vidb.client.VidbClient.getAccessTokenByClientCredentials] - Failed to fetch token by client credentials. Error: Request failed, returned error: HTTP/1.1 401 Unauthorized com.vmware.vrops.vidb.client.exception.VidbClientAuthException: Request failed, returned error: HTTP/1.1 401 Unauthorized
 at com.vmware.vrops.vidb.client.VidbClient.executeRequest(VidbClient.java:545) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vrops.vidb.client.VidbClient.getAccessTokenByClientCredentials(VidbClient.java:301) ~[vrops-vidb-client-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.auth.vidb.task.VidbClientSecretRotationTask.run(VidbClientSecretRotationTask.java:42) ~[vcops-auth-server-1.0-SNAPSHOT.jar:?]
 at com.vmware.vcops.platform.distributedtask.DistributedTaskExecutor$TaskProcessorThread.run(DistributedTaskExecutor.java:576) ~[alive_platform.jar:?]
 at com.integrien.alive.common.util.BaseThread$BaseThreadRunnable.run(BaseThread.java:177) ~[vrops-adapters-sdk.jar:?]
 at java.lang.Thread.run(Unknown Source) ~[?:?]

Environment

VCF Operations 9.0.x
VMware Identity Broker 9.0.x

Cause

This issue originates from a network partition that creates a split-brain scenario between the Primary and Replica nodes. During this isolation, the original Primary node successfully rotates the internal VIDB client secret. However, when the partition heals and the cluster rejoins, the system prioritizes the database state of the promoted Replica node, which still holds the old, unrotated secret. This action effectively overwrites the valid new secret with stale data. Consequently, once the old secret expires, the application attempts to authenticate using invalid credentials, resulting in permanent 401 Unauthorized errors.

Resolution

Note: Emphasize to take a snapshot of the VCF Operations nodes.

Follow the below steps to rotate the secret:

SSH as root to the primary node of VCF Operations.

Phase 1: Authentication & Discovery

Step 1: Fetch VCF SSO Auth Source ID
Retrieve all authentication sources to identify the VIDB source ID. Endpoint is unauthenticated.

curl -X 'GET' \
  'https://<OPS_HOST>/suite-api/api/auth/sources' \
  -H 'accept: application/json'

Step 2: Acquire Operations Token
Obtain an authentication token for the admin user. This token is required for all subsequent steps.

curl -X 'POST' \
  'https://<OPS_HOST>/suite-api/api/auth/token/acquire' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "username": "<ADMIN_USERNAME>",
  "password": "<ADMIN_PASSWORD>"
}'

Note: Save the token value from the response. It will be used as <OPS_TOKEN> in headers below.

Step 3: Retrieve Current Auth Source Configuration
Fetch the existing configuration. Modify the following JSON structure for the final step.

curl -X 'GET' \
  'https://<OPS_HOST>/suite-api/api/auth/sources/<AUTH_SOURCE_ID>' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>'

Phase 2: Secret Rotation

Before patching the source, generate a new secret for the VCF Ops Client.

Step 4: Identify the VIDB Resource ID
Identify the internal ID of the VIDB based on the hostname.

curl -X 'GET' \
  'https://<OPS_HOST>/suite-api/internal/vidb/vidbs' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>' \
  -H 'x-vrealizeops-api-use-unsupported: true'

Incase if the above command fails with an error " {"type":"Error","message":"Query Parameter \"vcfId\" is required.","httpStatusCode":400,"apiErrorCode":1509}" , you need to include the vcfId as a query parameter, as shown below:

curl -X 'GET' \
  'https://<OPS_HOST>/suite-api/internal/vidb/vidbs?vcfId=<vcf-instance-id>' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>' \
  -H 'x-vrealizeops-api-use-unsupported: true'

Note: <vcf-instance-id> can be found from the adapters API. Here are the steps:

Login to VCF Operations UI with admin privileges
Go to </> Developer Center > APIs & SDKs > Operations API, then launch the "API documentation"
Authorize the page using "admin" or equivalent permissions
Navigate to Adapters > GET /api/adapters > Click on "Try it out", then select Execute
In the response body output, check for "VMware Cloud Foundation Adapter" and note down the "id"
Use this id as the <vcf-instance-id> in the above query parameter

Step 5: Rotate Secret for the VCF Ops Client
Execute the rotation.

Target Client ID: Identified from the Logs.

Note: If the <CLIENT_ID> can't be identified from the logs, you can manually identify it by following these steps:

SSH to vCenter as root user.

Run the following API to get VC Session:

curl -k --request POST --url https://<MGMT VC FQDN>/rest/com/vmware/cis/session -u '<Username>:<password>'

#Note: Here the username is [email protected]

Run the following API to get tenant admin client token:

curl -k --location --request GET 'https://<MGMT VC FQDN>/api/vcenter/identity/broker/tenants/CUSTOMER/admin-client' --header 'vmware-api-session-id: <session id from step 1>' | jq

Run the following API to get VC's provider info and note down the "client_id" field from the below output:

curl -k --request GET \
  --url https://<MGMT VC FQDN>/api/vcenter/identity/providers/CUSTOMER \
  --header 'content-type: application/json' \
  --header 'vmware-api-session-id: <session from step 1>'

Resource ID: Use the ID found in Step 4.

curl --request PUT \
  'https://<OPS_HOST>/suite-api/internal/vidb/oauthclients/<CLIENT_ID>/secrets/rotate?vidbResourceId=<RESOURCE_ID>' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>' \
  -H 'x-vrealizeops-api-use-unsupported: true'

Critical: The API response will return the new client secret. Copy this value immediately for use in the final step.

Phase 3: Apply Configuration Patch

Step 6: Update Auth Source with New Credentials
Modify the configuration retrieved in Step 3 and apply the changes.

curl -X 'PUT' \
  'https://<OPS_HOST>/suite-api/api/auth/sources' \
  -H 'accept: application/json' \
  -H 'Authorization: OpsToken <OPS_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{
  "id": "<AUTH_SOURCE_ID>",
  "name": "VCF SSO",
  "sourceType": {
    "id": "VIDB",
    "name": "VIDB"
  },
  "created": 0,
  "lastModified": 0,
  "property": [
    {
      "name": "display-name",
      "value": "VCF SSO"
    },
    {
      "name": "issuer-url",
      "value": "https://<VCF_SSO_HOST>/acs/t/<TENANT_NAME>"
    },
    {
      "name": "client-id",
      "value": "<CLIENT_ID>"
    },
    {
      "name": "client-secret",
      "value": "<NEW_CLIENT_SECRET>"
    }
  ],
  "certificates": []
}'

Additional Information

Placeholder	Description
<OPS_HOST>	VCF Operations Manager hostname or IP
<ADMIN_USERNAME>	VCF Ops admin username
<ADMIN_PASSWORD>	VCF Ops admin password
<OPS_TOKEN>	Operations token from Step 2
<AUTH_SOURCE_ID>	Authentication source UUID from Step 1
<VCF_SSO_HOST>	VCF SSO server hostname
<TENANT_NAME>	SSO tenant name
<CLIENT_ID>	OAuth2 client ID
<CLIENT_SECRET>	OAuth2 client secret