ESXi upgrade in SDDC manager fails during the ESX_UPGRADE_VUM_STAGE_POSTCHECK stage

Products

• VMware Cloud Foundation

Issue/Introduction

• When upgrading ESXi hosts via SDDC Manager, the task fails during the ESX_UPGRADE_VUM_STAGE_POSTCHECK phase.
  
  
• In SDDC Manager, the /var/log/vmware/vcf/lcm/lcm.log file shows the following errors:  
  
  YYYY-MM-DD HH:MM:SS ERROR [vcf_lcm,########################,####] [c.v.v.i.b.i.TranslationMessage,http-nio-###.#.#.#-7400-exec-7] Can't find resource for bundle java.util.PropertyResourceBundle, key ESX_UPGRADE_VUM_POSTCHECK_FAILED_UNKNOWN_ERROR.remedy
YYYY-MM-DD HH:MM:SS ERROR [vcf_lcm,########################,####] [c.v.evo.sddc.lcm.model.task.SubTask,http-nio-##.#.#.#-7400-exec-7] Upgrade error occured:The ESXi update postcheck failed with an unknown error.
YYYY-MM-DD HH:MM:SS INFO,"upgradeStatus":"COMPLETED_WITH_FAILURE","upgradeError":{"errorType":"RECOVERABLE","stage":"ESX_UPGRADE_VUM_STAGE_POSTCHECK","errorCode":"ESX_UPGRADE_VUM_POSTCHECK_FAILED_UNKNOWN_ERROR","errorDescription":"\nThe ESXi update postcheck failed with an unknown error.","errorDescriptionCodes":["ESX_UPGRADE_VUM_POSTCHECK_FAILED_UNKNOWN_ERROR"],"errorDescriptionAttributes":{"0":"Existing SSH key not found for host <hostname>"},"metadata":"\nManual intervention needed as upgrade failed during install stage.\nCheck for errors in the lcm log files located on SDDC Manager under /var/log/vmware/vcf/lcm.log

Environment

VMware Cloud Foundation (VCF) 4.x

VMware Cloud Foundation (VCF) 5.x

Cause

SDDC Manager fails to authenticate with the ESXi host over SSH because the host key is either missing, mismatched, or the known_hosts file is improperly formatted.

Resolution

1. Establish an SSH session to the SDDC Manager as vcf and escalate to root user.
   
   
2. Verify that the SDDC Manager can communicate to the ESXi host over port 22.
   
nc -vz esxi.example.com 22 
   
   
3. Run the fixHostKeys.py script to add the host keys to the SDDC Manager as per the KB: How to update the SSH host keys on the SDDC Manager
   
   
4. Restart the management services to apply the changes: systemctl restart lcm systemctl restart commonsvcs


5. Navigate to the SDDC Manager UI and click Retry on the failed upgrade task.