Error "Step vmsp_upgrade failed" when upgrading VCF Services Runtime from 9.1 to 9.1.1 due to expired svc-vcf-sddc-lcm-VCENTER service account password
search cancel

Error "Step vmsp_upgrade failed" when upgrading VCF Services Runtime from 9.1 to 9.1.1 due to expired svc-vcf-sddc-lcm-VCENTER service account password

book

Article ID: 459346

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

  • VCF Services Runtime component upgrade from version 9.1.0 to 9.1.1 fails with the error message: "Step vmsp_upgrade failed. Reference Code: 9915FE3F. Check logs for details."
  • Running kubectl get workflow -A on the VCF Services Runtime Control Plane shows the upgrade-component, vcenter-role-sync, and configure-component workflows in a Failed state with the message retryStrategy.expression evaluated to false.

    • kubectl get workflow -A

      NAMESPACE       NAME                                             STATUS      AGE   MESSAGE
      vmsp-platform   configure-component-##############               Failed      ##h   retryStrategy.expression evaluated to false
      vmsp-platform   logging-configuration-precheck-#####             Failed      ##h   No more retries left
      vmsp-platform   upgrade-component-###########                    Failed      ##h   retryStrategy.expression evaluated to false
      vmsp-platform   vcenter-role-sync-######                         Failed      ##h   retryStrategy.expression evaluated to false
      vmsp-platform   wal-s3-cleanup-#########                         Succeeded   ##m

  • NOTE: Steps to SSH to the VMSP control plane node: 
    • Open the VCF Operations UI and navigate to the Build > Lifecycle > Components tab.
    • Click on the VCF Services Runtime link.
    • Scroll down the page to the Nodes section.
    • Identify a Control Plane node and take note of the IP address.
    • SSH to this Control Plane node as vmware-system-user.

  • Describing the failed upgrade-component workflow shows task contention errors:

    • kubectl describe workflow -n vmsp-platform upgrade-component-###########

      time="YYYY-MM-DDTHH:MM:SS" level=info msg="0/1 success conditions matched"
      time="YYYY-MM-DDTHH:MM:SS" level=info msg="Waiting for resource workflow.argoproj.io/vcenter-role-sync-##### in namespace vmsp-platform resulted in retryable error: Neither success condition nor the failure condition has been matched. Retrying..."
      time="YYYY-MM-DDTHH:MM:SS" level=info msg="failure condition '{status.phase in [Error Failed]}' evaluated true"

  • The vcenter-role-sync pod logs explicitly indicate an authentication failure for the SDDC LCM service account:

    • kubectl logs -n vmsp-platform <vcenter-role-sync-pod>

      YYYY/MM/DD HH:MM:SS INFO : hycqwemo############ Found admin credentials for user: svc-vcf-sddc-lcm-VCENTER-<id>@vsphere.local
      YYYY/MM/DD HH:MM:SS INFO : hycqwemo############ Target version: not specified
      YYYY/MM/DD HH:MM:SS INFO : hycqwemo############ vCenter server: <vCenter_Server_FQDN>
      YYYY/MM/DD HH:MM:SS INFO : hycqwemo############ Attempting secure certificate validation for vCenter connection
      YYYY/MM/DD HH:MM:SS INFO : hycqwemo############ Certificate validation enabled using known_hosts file at /tmp/govc_known_hosts
      govc: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
      YYYY/MM/DD HH:MM:SS ERROR : hycqwemo############ Failed to connect to vCenter server <vCenter_Server_FQDN>
      YYYY/MM/DD HH:MM:SS ERROR : hycqwemo############ Script exited unexpectedly at line 1 with exit code 1

  • The sddc-build-service pod logs confirm the same InvalidLogin exception for the account svc-vcf-sddc-lcm-VCENTER-<id>@vsphere.local

    • kubectl logs vcf-sddc-build-service-sddcbuild-######## -n vcf-sddc-lcm

      YYYY-MM-DDTHH:MM:SS ERROR sddc-build-service 44 [lcm@4413 scPath="ops" thread="scheduled-task-pool-2" logger="c.b.vcf.vsphere.client.VsphereClient" method="<init>:141"] [] - Failed to connect to https://<vCenter_Server_FQDN>:443/sdk as svc-vcf-sddc-lcm-VCENTER-<id>@vsphere.local java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {

Environment

VCF Operations 9.1

Cause

This issue occurs because the vSphere local service account (svc-vcf-sddc-lcm-VCENTER-<id>@vsphere.local) used by the VCF Services Runtime for SDDC lifecycle management operations has an expired password. This prevents the runtime components from authenticating to the vCenter Server SDK during the upgrade workflows.

Resolution

To resolve this issue, regenerate the vCenter service account and update the Kubernetes secret within the VCF Services Runtime environment.

  1. Download the Commonutil.sh and ReestablishVCServiceAccount.sh scripts from the following KB article: Service Account Recovery Script For vCenter and SDDC Manager for VMware Cloud Foundation and VMware vSphere Foundation to the VCF Services Runtime Control Plane node under a directory which should be accessible by the root user.
  2. Make the scripts executable:
    chmod +x Commonutil.sh
    chmod +x ReestablishVCServiceAccount.sh
  3. Run the script, providing the vCenter Server FQDN and the vCenter Server's administrator account credentials:
    ./ReestablishVCServiceAccount.sh --vcenter-fqdn '<vcenter-fqdn>' --vcenter-admin-username '[email protected]'
  4. Verify the script completes successfully and outputs "Service account recovery completed successfully".
  5. Attempt the VCF Services Runtime upgrade again from VCF Operations UI.
Powered by Wolken Software