ESXi update fails with error "Upgrade is requested for cluster <Cluster_ID> which is in failed state" due to Lockdown mode.

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When performing SDDC-managed updates on ESXi hosts in VMware Cloud Foundation (VCF), the following error message is observed: "Upgrade is requested for cluster <Cluster_ID> which is in failed state"
• ESXi host update fails during the precheck.
• SDDC Password manager shows the service account disconnected and root account "--".
• The ESXi service account (svc-vcf-<host_shortname>) shows a Disconnected status in the SDDC Manager dashboard.

• /var/log/vmware/vcf/lcm/lcm.log will contain the log snippets for password validation precheck failing. 

  YYYY-MM-DDTHH:MM INFO  [vcf_lcm,69ba################2b5a3,c370] [c.v.e.sddc.lcm.audit.AuditScheduler,Scheduled-3] Last upgrade status for software type ESX_HOST and resourceId 1051d074-####-####-####-########e5f8 is: {"id":"2621","bundleId":"9e494f50-####-####-####-#######eed3","bundleElementId":"4e5d3fc2-####-####-####-de4de361c585","bundleElementOrder":1,"resourceType":"ESX_HOST","resourceId":"1051d074-####-####-####-fd9af01be5f8","upgradeId":"1de183b8-####-####-####-7d9baf96fc30","upgradeStatus":"COMPLETED_WITH_FAILURE","upgradeError":{"errorType":"RECOVERABLE","stage":"ESX_UPGRADE_VUM_STAGE_PRECHECK","errorCode":"com.vmware.vcf.error.esx.password.validation.precheck.execution.failed","errorDescription":"Password validation execution failed due to <Host_FQDN>: Validation of password of ESXi host : Host_FQDN has failed.

• /var/log/vmware/vcf/operationsmanager/operationsmanager.log reports the Password validation cannot be performed when the host is in lockdown mode.

  YYYY-MM-DDTHH:MM DEBUG [vcf_om,69aa################a546,df15] [c.v.v.p.s.PasswordExpirationService,om-exec-2] Expiry retrieval status : UNKNOWN ,  Diagnostic message : {"errorCode":"PASSWORD_VALIDATE_RESOURCE_NOT_SUPPORTED","arguments":["<Host_FQDN>"],"errorMessage":"Password validation cannot be performed when host <Host_FQDN> is in lockdown mode.","referenceToken":"######"}

• Lockdown Mode is enabled on the ESXi host, but the required VCF service account (e.g., svc-vcf-<host_shortname>) has not been added to the Exception Users list.

Environment

• VCF 5.x

Cause

When Lockdown Mode is active, direct authentication to the ESXi host is disabled for all users except those explicitly named in the Exception Users list. Without this exception, any management operation like updates performed on the host is blocked. This prevents SDDC Manager from gaining the necessary permissions to manage the host during the update process.

Resolution

1: Verify Service Account: Ensure the "svc-vcf-<host_shortname>" account exists on the host and has the Administrator role.

2: Configure Exception List:
  ○ Navigate to the host in the vSphere Client.
  ○ Go to Configure > System > Security Profile.
  ○ Under Lockdown Mode, click Edit and add the service account "svc-vcf-<host_shortname>" to Exception Users.

3: If the svc-vcf account is missing from the host, manually re-create it and wait for SDDC password manager to resync. Refer: Re-create missing SDDC Manager Service account on an ESXi host 

4: Depending on the service account status in the SDDC password manager, take next steps:

4a: Service account status reaches active state in the SDDC password manager: Proceed to step 5.

4b: Service account status still continues to be in disconnected state:

• • Retrieve the password (secret) stored for the ESXi service account in the SDDC password manager: KB: Retrieve the service accounts credentials from SDDC Manager 

Example:
   "creationTime" : 173807#######,
      "credentialType" : "SSH",
      "entityId" : "18aa####-####-####-####-########d1da",
      "entityType" : "ESXI",
      "id" : "3995####-####-####-####-########47f3",
      "modificationTime" : 173807#######,
      "secret" : "####################",
      "serviceId" : "012b####-####-####-####-########d230",
      "serviceType" : "SDDC_MANAGER",
      "targetType" : "ESXI",
      "username" : "svc-vcf-<host_shortname>"

• • Update the svc-vcf-<host_shortname> account password in the ESXi host with the credentials retrieved ("secret" : "####################") and save the changes.
  • Clear password lockout for the service account by running the following command in ESXCLi:  pam_tally2 -u svc-vcf-<host_shortname> -r 
  • Remediate the password for svc-vcf-<host_shortname> account in the SDDC manager with the previously obtained password (secret). Refer: Remediate Passwords

5: If there are hosts in ERROR status in the SDDC hosts inventory, change its status to ACTIVE.

  a. Take snapshot of the SDDC manager appliance without virtual macine memory.

  b. Connect SSH to the SDDC manager.

  c. Run the following command to check platform DB's "host" table to ensure there are no hosts in ERROR state:

# psql -h localhost -U postgres -d platform -c "select id,hostname,status from host where status!='ACTIVE'"

  d. If there are hosts in ERROR state as seen in the below example output, change the host configuration status to ACTIVE.

root@<hostname> [ /home/vcf ]# psql -h localhost -U postgres -d platform -c "select id,hostname,status from host where status!='ACTIVE'"
                  id                  |  hostname   | status
--------------------------------------+-------------+--------
 1051####-####-####-####-########e5f8 | <Host_FQDN> | ERROR

  e. Update the problematic host status to Active by using the following command:

UPDATE host set status = 'ACTIVE' where id = '<host-id>';

Example: UPDATE host set status = 'ACTIVE' where id = '1051####-####-####-####-########e5f8';

6: Proceed to run pre-upgrade check and update ESXi hosts via SDDC manager.