VCF Services Runtime authentication fails due to an expired or deleted vCenter service account

Products

• VCF Automation

Issue/Introduction

• On the VCF Operations UI - returns a health check error for the vCenter synthetic check platform-vc-serviceaccount-http 
• Any Day N operations on VCF Management Services or VCF Automation deployment that require vCenter orchestration (e.g.,Volume Resize, Node Rollout, Scale up or Scale out of the component) fail with authentication errors

Environment

VCF Services Runtime 9.1.0
VCF Automation 9.0.0 and 9.1.0

Cause

This issue occurs when the dedicated service account utilized by the VCF Services Runtime to authenticate with vCenter has either expired or been deleted.

Resolution

To resolve this issue, you must run the attached remediation script using vCenter administrator credentials. This script fixes the vCenter service account utilized by the VCF Services Runtime.
You will need direct SSH access to a VCF Services Runtime Control Plane node to perform these steps.

Prerequisites

1. Download the service-account-remediation.sh script attached to this article.

2. Ensure you have your vCenter administrator credentials on hand.

3. Identify the IP address of a Control Plane node:

   • Log in to the VCF Operations UI.

   • Navigate to Build > Lifecycle > Components.

   • Click the VCF Services Runtime link.

   • Scroll down to the Nodes section.

   • Locate a Control Plane node and record its IP address.

Step 1: Copy the remediation script to the control plane node.

From your local machine, securely copy the downloaded script to the vmware-system-user home directory on the node. Replace <node-ip> with the IP address identified in the prerequisites:

scp service-account-remediation.sh vmware-system-user@<node-ip>:service-account-remediation.sh
Note: You will be prompted to enter the vmware-system-user account password.

Step 2: Connect to the cluster node and initialize access

1. 1. 1. SSH to this Control Plane node as vmware-system-user and provide the password for the account.
      2. Elevate to root for obtaining keys to the kubernetes layer of the VCF Service Runtime:
         
         sudo -i
         export KUBECONFIG=/etc/kubernetes/admin.conf

Step 3: Execute the remediation script

chmod +x /home/vmware-system-user/service-account-remediation.sh
/home/vmware-system-user/service-account-remediation.sh

Upon execution, the script will prompt you for vCenter administrator credentials. It utilizes this account to automatically repair the vCenter service account associated with the VCF Services Runtime.

Additional Information

VCF Services Platform Cluster Health Checks