VMware Cloud Foundation upgrade fails while putting an ESXi host into maintenance mode

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• When performing a VMware Cloud Foundation upgrade, the upgrade process fails while waiting to put an ESXi host into maintenance mode.
• There are either a large number of virtual machines being migrated or a large amount of vSAN data being moved as a result of the maintenance mode operation.

Environment

VMware Cloud Foundation 2.1.x
VMware Cloud Foundation 2.0.x
VMware Cloud Foundation 2.3.x
VMware Cloud Foundation 2.2.x

Resolution

This is a known issue affecting VMware Cloud Foundation. This issue issue is less likely to occur in versions 2.2 and beyond.

Workaround:

To work around the issue, increase the amount of time that the LCM process will wait for a host to complete its upgrade operation.
 

For VMware Cloud Foundation 2.1.x:

1. Log in to the VRM virtual machine as the root user.
2. Make a copy of the /home/vrack/lcm/lcm-app/conf/application-prod.properties file:
   
   cp /home/vrack/lcm/lcm-app/conf/application-prod.properties /tmp/
    
3. Open the /home/vrack/lcm/lcm-app/conf/application-prod.properties file using a text editor:
   
   vi /home/vrack/lcm/lcm-app/conf/application-prod.properties
    
4. Find the line that reads:
   
   lcm.upgrade.esx.maxallowed.increase.endTime=14400000
   
   and change 14400000 to a higher value.
   
   Note: 14400000 corresponds to four hours.
    
5. Save and close the file.
6. Restart the lcm-init service:
   
   service lcm-init restart

 

For VMware Cloud Foundation 2.2.x and higher:

1. Log in to the SDDC Manager Controller virtual machine as the root user.
2. Make a copy of the /home/vrack/lcm/lcm-app/conf/application-evo.properties file:
   
   cp /home/vrack/lcm/lcm-app/conf/application-evo.properties /tmp/
    
3. Open the /home/vrack/lcm/lcm-app/conf/application-evp.properties file using a text editor:
   
   vi /home/vrack/lcm/lcm-app/conf/application-evo.properties
    
4. Find the line that reads:
   
   esx.upgrade.timeout=14400000
   
   and change 14400000 to a higher value.
   
   Note: 14400000 corresponds to four hours.​
5. Save and close the file.
6. Restart the lcm service:
   
   systemctl restart lcm

Note: The preceding steps may not resolve the issue and manual intervention on the ESXi host(s) in question may be required. It may be necessary to reboot the host.

Additional Information

Settings which can be tuned to prevent timeouts during upgrades in VMware Cloud Foundation