Upgrading NSX-T from SDDC Manager fails due to an ESXi host not entering in maintenance mode

Products

• VMware SDDC Manager

Issue/Introduction

Symptoms:

• From /var/log/vmware/vcf/lcm/lcm-debug.log:
  yyyy-mm-ddThh:mm:ss.xxx+0000 ERROR [vcf_lcm,############-####-###########,###] [c.v.evo.sddc.lcm.model.task.SubTask,http-nio-###.#.#.#-####-exec-1] Upgrade error occurred: <ESXi_fqdn> in workload do main <WLD_Name> of cluster <Cluster_Name> cannot enter maintenance mode as VM cannot migrate. Generic error : The operation is not allowed in the current state. <ESXi_fqdn> in workload domain  <WLD_Name> of cluster <Cluster_Name> cannot enter maintenance mode, as VM cannot migrate. Generic error : The operation is not allowed in the current state. Reference token #####
  
  
• SDDC Manager subtasks:
  Message: Host group upgrade status is FAILED for group ############-####-###########:domain-cx <ESXi_fqdn> : [Upgrade failed: Virtual machine 'VM_Name', located on host '<ESXi_fqdn>', reported an issue which prevents entering maintenance mode: The operation is not allowed in the current state., Upgrade failed: Virtual machine 'VM_Name', located on host '<ESXi_fqdn>', reported an issue which prevents entering maintenance mode: Failed to move a VM out of maintenance mode candidate host since the VM is in manual mode., Upgrade failed: DRS reported issues that prevent host '<ESXi_fqdn>l' from entering maintenance mode.].
Remediation Message: Re-run the upgrade precheck (resolve any issues). Retry the upgrade, once available.
Reference Token: ######

Environment

VMware Cloud Foundation 5.x

Cause

The problem is caused by some VMs not migrating automatically due to an underlying issue.

Resolution

1. Troubleshoot and fix the vmotion issues on the VMs. 
2. Restart the upgrade again from the SDDC manager UI.

Note: If the VMs aren't critical, power off the impacted VMs and migrate them to another ESXi host as a workaround. 

 

Additional Information

Reference article: Understanding and troubleshooting vMotion