Error: 401 Unauthorized during VCF Operations upgrade precheck stage ops_stage_precheck

Products

• VCF Operations

Issue/Introduction

When attempting to upgrade VMware Cloud Foundation (VCF) components, the upgrade precheck fails during the VCF Operations stage.

UI Error: An unexpected error occurred in step ops_stage_precheck. Reference Code: 1E3320C6.

Error Detail: 401 Unauthorized

Log Observations: In the vcf-sddc-lcm logs, authentication attempts to the VCF Operations API (/suite-api/api/auth/token/acquire) using the lcm-ops user fail with an HTTP 401 response, despite valid credentials existing previously.

Environment

• VMware Cloud Foundation (VCF) 9.1.x
• VCF Operations

Cause

The credentials for the lcm-ops service account have fallen out of sync between VCF Fleet Lifecycle Manager (LCM) and the VCF Operations instance. This typically occurs due to manual password policy changes (e.g., minimum length increases), manual password rotations that did not propagate, or account lockouts within the Operations local auth source.

Resolution

Note: Check the vCenter inventory for a 9.0 Fleet Management Appliance, and ensure it is not powered on. For more information see Upgrade prechecks fail with HTTP 401 Unauthorized when upgrading from 9.1.0.0#00 to 9.1.0.0#00 in VCF Operations.

To resolve this issue, perform a manual re-import of the VCF Operations component into Fleet LCM. This process refreshes the service account integration and restores authentication.

Step 1: Obtain VSP Token

SSH into the VSP Platform appliance as vmware-system-user.

Run the following command to set your environment variables and retrieve an access token:

VSP_PLATFORM_FQDN=<VSP_PLATFORM_FQDN>
FLEET_FQDN=<FLEET_FQDN>
VSP_PASSWORD=$(systemd-ask-password)


# Get VSP token
VSP_TOKEN=$(curl -ks --request POST \
  --url "https://$VSP_PLATFORM_FQDN/api/v1/identity/token" \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data grant_type=password \
  --data "[email protected]" \
  --data "password=$VSP_PASSWORD" \
  | jq -r '.access_token')

echo "VSP_TOKEN=${VSP_TOKEN}"

Step 2: Retrieve Component Information

Run the following to identify the sddcLcmId for the Operations component:

# Get Fleet LCM components
curl -sS -k "https://${FLEET_FQDN}/fleet-lcm/v1/components" \
  -H "Authorization: Bearer ${VSP_TOKEN}" \
  -H "Content-Type: application/json" | jq

Step 3: Perform Re-import

Create a file named /tmp/spec.json with the following structure, filling in the sddcLcmId, fqdn, and password retrieved from your environment:

# Update import spec if needed (sddcLcmId, certificate)
{
  "componentSpecs": [
    {
      "deploymentType": "ComponentImportSpec",
      "sddcLcmId": "<REPLACE_WITH_SDDC_LCM>",
      "componentType": "OPS",
      "fqdn": "<REPLACE_WITH_OPS_FQDN>",
      "username": "admin",
      "password": "<REPLACE_WITH_OPS_ADMIN_PASSWORD>",
      "certificate": "<REPLACE_WITH_OPS_CERT>"
    }
  ]
}

Trigger the re-import via the API:

curl -sS -k -X POST "https://${FLEET_FQDN}/fleet-lcm/v1/components" \
  -H "Authorization: Bearer ${VSP_TOKEN}" \
  -H "Content-Type: application/json" \
  -d @/tmp/spec.json | jq

Once the task completes, return to the VCF UI and retry the upgrade precheck.