Step vmsp_upgrade failed or InvalidLogin due to stale service account credentials in VMware Cloud Foundation and VMware vSphere Foundation

Products

• VMware vSphere Foundation
• VMware Cloud Foundation

Issue/Introduction

• In VMware vSphere Foundation (VVF) and VMware Cloud Foundation (VCF) environments, service account credentials stored in SDDC-LCM Kubernetes secrets become stale, causing workflow failures.
  
  
• During the upgrade of VCF Management, the VCF Operations and VCF Operations for Networks components failed to complete the post_upgrade_inventory_sync task with specific reference codes (e.g., BD974893).
  
  
• The VMSP upgrade fails with the generic error: Step vmsp_upgrade failed.
  
  
• The vcenter-role-sync step fails and exits.
  
  
• VMware Aria Automation 8.18.1 to VCF 9.x upgrades fail at Create Automation Cluster → Deploy VCF services runtime cluster with the error: Failed to obtain SDDC Manager token for 'https://<SDDC_FQDN>'. Cause: (empty).
  
  
• sddc-build-service logs show repeated authentication failures:
  

  Failed to connect to https://<vCenterFQDN>:443/sdk as svc-vcf-sddc-lcm-VCENTER-###@vsphere.local caused by vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.

• SDDC Manager commonsvcs logs display PERMISSION_NOT_FOUND followed by a SOAP fault.
  
  
• Repeated 401 Unauthorized errors trigger an SDDC Manager lockout, displaying CLIENT_IP_BLOCKED (403) on the runtime nodes.

Environment

• VMware Cloud Foundation (VCF)

• VMware vSphere Foundation (VVF)

• VMware Aria Automation

Cause

The issue arises from different credential management patterns across deployment types:

• VVF: SDDC-LCM is configured with [email protected] credentials instead of dedicated service accounts, preventing normal rotation workflows from functioning correctly. If the administrator password is rotated out-of-band, the stored credentials become invalid.

• VCF: Multiple service account rotations (scheduled or on-demand) delete the account (e.g., svc-vcf-sddc-lcm-SDDC_MANAGER-##### or svc-vcf-sddc-lcm-VCENTER-#####) from SDDC Manager or vCenter. This breaks the connection between SDDC-LCM and the target service, leaving the stored credentials stale and potentially causing IP lockouts.

Resolution

Use Cases

The recovery tooling is split into two dedicated scripts, one per target service. Each script creates a new service account that replaces the stored credentials in SDDC-LCM. Both handle the following use cases:

• Administrator credential or service account credential stored in SDDC-LCM is valid and in sync with the target service (vCenter/SDDC Manager).
• Administrator credential or service account credential stored in SDDC-LCM has been rotated out-of-band and is no longer in sync with the target service.

The scripts do not handle the repair of a corrupted administrator account. Instead, as part of a validation step, each script validates the passed administrator credentials. If the credentials are invalid, an error message indicates this.

Scripts

• CommonUtils.sh - Supporting functions called by other scripts. Ensure this is in the same working directory as the Reestablish scripts below.
• ReestablishVCServiceAccount.sh - Creates a vCenter service account with global Administrator permissions.
• ReestablishSddcManagerServiceAccount.sh - Creates an SDDC Manager service account with Admin role permissions.

ReestablishVCServiceAccount.sh Workflow

Validations

• Connectivity check to vCenter
• Credentials check for vCenter admin
• vCenter version check - requires 9.1 or later

Service Account Creation

• Creates a vCenter service account and assigns global Administrator permissions
• Validates the newly created service account credentials

Kubernetes Secret Patching

• Patches kubernetes secret vcf-iam-vcenter-admin with updated service account username and password

ReestablishSddcManagerServiceAccount.sh Workflow

Validations

• Connectivity check to SDDC Manager
• Credentials check for SDDC Manager admin

Service Account Creation

• Creates an SDDC Manager service account and assigns Admin role permissions
• Validates the newly created service account credentials

Kubernetes Secret Patching

• Patches kubernetes secret vcf-iam-sddc-manager-admin with updated service account username and password

ReestablishVCServiceAccount.sh Parameters

• --vcenter-fqdn: Fully Qualified Domain Name of the vCenter Server. Required to establish a connection to vCenter for service account creation, permission assignment, SSL certificate extraction, and configuration validation. All vCenter operations depend on this endpoint.
• --vcenter-admin-username: Administrator username for vCenter authentication. Required to authenticate with vCenter for creating new service accounts and assigning global Administrator permissions. Must have sufficient privileges to create users and modify permissions.

ReestablishSddcManagerServiceAccount.sh Parameters

• --sddc-manager-fqdn: Fully Qualified Domain Name of the SDDC Manager. Required to establish a connection to SDDC Manager for service account creation, role assignment, SSL certificate extraction, and configuration validation. All SDDC Manager operations depend on this endpoint.
• --sddc-manager-username: Administrator username for SDDC Manager authentication. Required to authenticate with SDDC Manager for creating new service accounts and assigning Admin role permissions. Must have sufficient privileges to create service accounts.

Resolution

This script creates new dedicated service accounts to replace stored administrator credentials, supporting both vCenter and SDDC Manager recovery scripts.

Prerequisites

• Terminal access with connectivity to the target services.
• Valid vCenter administrator credentials (for vCenter) or SDDC Manager admin credentials (for SDDC Manager).
• Must be run as root from a VMSP control plane node.
• Dependencies on: curl, openssl, jq, awk, base64, kubectl

Installation

Download all the scripts attached to this article:

Steps to SSH to the VMSP control plane node:

1. Open the VCF Operations UI and navigate to the Build > Lifecycle > Components tab.
2. Click on the VCF Services Runtime link.
3. Scroll down to the Nodes section and take note of a Control Plane node IP address.
4. SSH to this Control Plane node as vmware-system-user.
5. Switch to root and export the Kubernetes configuration:

   sudo su -
   export KUBECONFIG=/etc/kubernetes/admin.conf

6. Ensure kubectl is working and connected to the Control Plane node:

   kubectl get nodes -o wide

   Note: This command must complete successfully and return a list of nodes before you continue.

7. Create a directory and download the appropriate raw .sh files attached to this article (including CommonUtils.sh):

   mkdir recovery_scripts
   cd recovery_scripts

8. Make the scripts executable:

   chmod +x *.sh

9. Run the recovery script for your specific use case. If /tmp on the runtime nodes is mounted as noexec, bypass it by invoking bash directly:

   For vCenter:

   bash ./ReestablishVCServiceAccount.sh \
     --vcenter-fqdn '<VCENTER_FQDN>' \
     --vcenter-admin-username '[email protected]'

   For SDDC Manager:

   bash ./ReestablishSddcManagerServiceAccount.sh \
     --sddc-manager-fqdn '<SDDC_MANAGER_FQDN>' \
     --sddc-manager-username '[email protected]'

10. Restart the SDDC-LCM deployments to ensure they read the newly patched Kubernetes secrets:

    kubectl rollout restart deployment vcf-sddc-build-service-sddcbuild -n vcf-sddc-lcm
    kubectl rollout restart deployment vcf-sddc-upgrade-service-sddcupgrade -n vcf-sddc-lcm

11. If you experienced IP lockouts, restart commonsvcs on SDDC Manager to clear the in-memory blocks:

    systemctl restart commonsvcs