Service Account Recovery Script For vCenter and SDDC Manager for VMware Cloud Foundation and VMware vSphere Foundation

Products

• VMware vSphere Foundation
• VMware Cloud Foundation

Issue/Introduction

In VMware vSphere Foundation (VVF) and VMware Cloud Foundation (VCF) environments, service account credentials stored in SDDC-LCM Kubernetes secrets may become stale, causing workflow failures. This occurs in several scenarios:

• VVF environments: vCenter may be configured with break-glass [email protected] credentials instead of proper service accounts. If the administrator password is rotated out-of-band, the stored credentials become invalid.
• VCF environments: If SDDC Manager service account rotation is performed two or more times (scheduled or on-demand), the credentials may be deleted from SDDC Manager, causing stored credentials to become stale.

Cause

The issue arises from different credential management patterns across deployment types:

• VVF: SDDC-LCM is always configured with [email protected] credentials instead of dedicated service accounts, preventing normal rotation workflows from functioning correctly.
• VCF: Multiple service account rotations can result in credential deletion from SDDC Manager, breaking the connection between SDDC-LCM and the target service.

Resolution

Use Cases

The recovery tooling is split into two dedicated scripts, one per target service. Each script creates a new service account that replaces the stored credentials in SDDC-LCM. Both handle the following use cases:

• Administrator credential or service account credential stored in SDDC-LCM is valid and in sync with the target service (vCenter/SDDC Manager).
• Administrator credential or service account credential stored in SDDC-LCM has been rotated out-of-band and is no longer in sync with the target service.

The scripts do not handle the repair of a corrupted administrator account. Instead, as part of a validation step, each script validates the passed administrator credentials. If the credentials are invalid, an error message indicates this.

Scripts

• CommonUtils.sh - Supporting functions called by other scripts. Ensure this is in the same working directory as the Reestablish scripts below.
• ReestablishVCServiceAccount.sh - Creates a vCenter service account with global Administrator permissions.
• ReestablishSddcManagerServiceAccount.sh - Creates an SDDC Manager service account with Admin role permissions.

ReestablishVCServiceAccount.sh Workflow

Validations

• Connectivity check to vCenter
• Credentials check for vCenter admin
• vCenter version check - requires 9.1 or later

Service Account Creation

• Creates a vCenter service account and assigns global Administrator permissions
• Validates the newly created service account credentials

Kubernetes Secret Patching

• Patches kubernetes secret vcf-iam-vcenter-admin with updated service account username and password

ReestablishSddcManagerServiceAccount.sh Workflow

Validations

• Connectivity check to SDDC Manager
• Credentials check for SDDC Manager admin

Service Account Creation

• Creates an SDDC Manager service account and assigns Admin role permissions
• Validates the newly created service account credentials

Kubernetes Secret Patching

• Patches kubernetes secret vcf-iam-sddc-manager-admin with updated service account username and password

ReestablishVCServiceAccount.sh Parameters

• --vcenter-fqdn: Fully Qualified Domain Name of the vCenter Server. Required to establish a connection to vCenter for service account creation, permission assignment, SSL certificate extraction, and configuration validation. All vCenter operations depend on this endpoint.
• --vcenter-admin-username: Administrator username for vCenter authentication. Required to authenticate with vCenter for creating new service accounts and assigning global Administrator permissions. Must have sufficient privileges to create users and modify permissions.

ReestablishSddcManagerServiceAccount.sh Parameters

• --sddc-manager-fqdn: Fully Qualified Domain Name of the SDDC Manager. Required to establish a connection to SDDC Manager for service account creation, role assignment, SSL certificate extraction, and configuration validation. All SDDC Manager operations depend on this endpoint.
• --sddc-manager-username: Administrator username for SDDC Manager authentication. Required to authenticate with SDDC Manager for creating new service accounts and assigning Admin role permissions. Must have sufficient privileges to create service accounts.

Resolution

This script creates new dedicated service accounts to replace stored administrator credentials, supporting both vCenter and SDDC Manager recovery scripts.

Prerequisites

• Terminal access with connectivity to target services
• Valid vCenter administrator credentials (for vCenter) or SDDC Manager admin credentials (for SDDC Manager)

The script requires the following:

• Must be run as root from a VMSP control plane node
• Dependencies on: curl, openssl, jq, awk, base64, kubectl

Installation

Download the appropriate script attached to this case depending on the use case:

Steps to SSH to the VMSP control plane node:

1. Open the VCF Operations UI and navigate to the Build > Lifecycle > Components tab.
2. Click on the VCF Services Runtime link.
3. Scroll down the page to the Nodes section.
4. Identify a Control Plane node and take note of the IP address.
5. SSH to this Control Plane node as vmware-system-user.
6. Run the following commands, providing the password as necessary:

sudo -i
export KUBECONFIG=/etc/kubernetes/admin.conf

Ensure that kubectl is working as expected and the SSH is connected to a Control Plane node by running the following command:

kubectl get nodes -o wide

Note: This kubectl command must complete successfully and return a list of nodes before continuing.

Download the raw files onto the VMSP control plane node using wget and run the following commands or use vi to copy in the logic to the new file:

mkdir recovery_scripts
cd recovery_scripts
wget ...
chmod +x *.sh

Run the recovery script for the given use case. The script prompts for the target service admin password and prompts to confirm the SSL thumbprint as needed.

Example Commands

vCenter

./ReestablishVCServiceAccount.sh \
  --vcenter-fqdn '<VCENTER_FQDN>' \
  --vcenter-admin-username '[email protected]'

SDDC Manager

./ReestablishSddcManagerServiceAccount.sh \
  --sddc-manager-fqdn '<SDDC_MANAGER_FQDN>' \
  --sddc-manager-username '[email protected]'