Setting the upgrade order for ESXi hosts during a VMware Cloud Foundation upgrade

Products

• VMware Cloud Foundation

Issue/Introduction

This article provides options to queue ESXi hosts in maintenance mode with an order that is suitable for your workloads. In SDDC Manager, there are two workflows that require ESXi hosts to enter maintenance mode:

1. NSX VIB upgrades
2. ESXi host upgrade

Environment

VMware Cloud Foundation 5.1.x
VMware Cloud Foundation 5.0.x
VMware Cloud Foundation 4.5.x

Resolution

Scope

This article contains a manual and an automatic approach to set the upgrade order of ESXi hosts. The two workflows are following the same path and they are interchangeable.

Manual Procedure

NSX VIB upgrade

To control the order in which ESXi hosts will enter maintenance mode during NSX VIB upgrade, you must first upgrade the NSX Upgrade Coordinator, all NSX Edge clusters, and optionally upgrade all vSphere Clusters which do not require special ordering. Once completed, ESXi host ordering can be achieved in the NSX UI, by following the procedure below.

Orchestrate the Order of VIB Upgrades in a VMware Cloud Foundation Environment

1. Start the NSX upgrade for a VI workload domain. Select one of the following options:

   • If you have NSX Edge clusters, select Upgrade only NSX Edge Clusters. This option will disable the ESXi host upgrade in the SDDC Manager wizard. Complete the wizard to initiate the upgrade. Once this is done you could upgrade all vSphere clusters that do not require strict ordering.
   • If you do not have NSX Edge clusters, select one or multiple vSphere clusters and complete the wizard to initiate the upgrade.

     Note You cannot control the host ordering in the selected clusters.

2. Once the upgrade initiated in Step 1 has finished successfully, log in the NSX Manager cluster by using the VIP address.

3. In the NSX Manager UI, navigate to System > Lifecycle Management > Upgrade, and click Continue with the Upgrade.

   Note: If you are redirected to the IP address or FQDN of an individual node, proceed to the UI of this node - it is the one that orchestrates the upgrade process.

4. If prompted, run pre-checks in the NSX Upgrade UI and check the result. Resolve any issues that have been flagged.

5. Go to the Hosts tab. At this point, you should plan the upgrade order. In the Plan section, select Serial or Parallel host group upgrade according to the type of storage, free resources, and individual needs.

   You can upgrade all ESXi hosts in a single group in parallel if all the hosts from the group can enter maintenance mode at the same time without violating any policy (e.g., vSAN availability, DRS rules., etc.). Use your regular procedure to enter ESXi hosts in a maintenance mode.

   Note: There is a check in the NSX VIB upgrade workflow: NSX does not allow two (or more) hosts from the same cluster to be placed in the same group if this group is set to Parallel mode.

   • You are required to run a sequential VIB upgrade in the same vSAN cluster.
   • For any other type of storage you may run VIB upgrade as a Parallel operation in the same group, if all ESXi hosts in the group can enter maintenance mode at the same time, or are already in maintenance mode.

6. Repeat the following steps until you have upgraded all VIBs in ESXi hosts in the NSX Manager inventory.

   1. Verify that all host groups are disabled. Otherwise, disable them.

   2. Create host groups (e.g. Upgrade Now_XX) for ordering ESXi hosts upgrade. Plan accordingly for your desired upgrade workflow.

   3. Move ESXi hosts to the newly created groups that will be upgraded.

   4. Verify that there are no warnings or errors in the vCenter Server for the ESXi hosts or clusters that will be upgraded. Resolve any issues before running the upgrade.

      • Verify cluster health and health of the ESXi hosts which will be upgraded.
      • Verify vSAN Skyline Health.

   5. Make sure that only the groups that you want to upgrade are in Enabled state and then click on Continue/Start to start the upgrade process.

   6. When the ESXi host upgrade is completed, create a group (e.g. 'Upgraded Hosts') that will contain upgraded hosts and move hosts that are in groups created in step "b" to this group.

7. After all ESXi hosts are upgraded, move hosts back to their original host groups and enable all original groups in the following way:

   1. Click Reset at the top of the Hosts page and then click Continue in the Reset Host Upgrade Plan confirmation dialog. This will revert all original groups and enable them.
   2. Make sure that the state of all groups is Enabled with a green circle.

8. After about 15 minutes, log in the SDDC Manager UI.

9. In SDDC Manager, start the NSX upgrade. At this point, the upgrade should start directly, without the usual upgrade wizard because the only pending upgrade is for NSX Managers.

10. After NSX Managers are upgraded through SDDC Manager, the UI shows vCenter Server as the next available component in the upgrade sequence. Continue with the upgrade as usual.

ESXi Upgrade

To control the order in which ESXi hosts will enter maintenance mode during ESXi Upgrade you can follow the "Skip Hosts During vSphere clusters Update" steps in the "VMware Cloud Foundation Lifecycle Management" documentation.

Note: You cannot skip hosts that are part of a VI workload domain that is using vSphere Lifecycle Manager images, since these hosts are updated at the cluster-level and not at the host-level.

Automation Procedure

In the attached file you can find a PowerShell script that can help you to perform some of the steps for ESXi upgrade ordering. Please refer to the README.pdf file in the archive for further information.