Update vSphere iSCSI vmk and iSCSI target/array network configuration and avoid All Paths Down (APD)

Products

• VMware vCenter Server

Issue/Introduction

A maintenance operation is being performed on the storage array that requires a subnet mask update across the entire iSCSI network. Because some storage arrays require the change to be applied globally rather than on individual uplinks, there can be a total loss of connectivity to the datastores during the transition. This article provides the procedure to properly decommission the datastores on the VMware side to prevent host instability and safely bring them back online after the network update.

Caveats:

• If the storage array allows for multiple subnets, a new vSphere vmkernel adapter can be created on the hosts to retain connectivity to the backing LUNs.
• The KB resolution steps assume there is another storage array to which VMs can be migrated.

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x

Cause

When a storage array or vSwitch undergoes a disruptive change (like a netmask update), the iSCSI sessions between the ESXi hosts and the targets can be broken. If the ESXi host is still attempting to perform I/O to a mounted datastore when the path is lost, it can trigger an All Paths Down (APD) condition. These states can cause the ESXi management agents to become unresponsive and may result in virtual machine hangs or crashes. Many storage arrays allow these changes to be made concurrently, but some arrays set this globally for the network and not on individual uplinks.

Resolution

Follow a multi-phase approach to ensure the hosts are aware that the storage loss is intentional.

Pre-Requisites

1. Evacuate the Datastore: Use Storage vMotion to migrate all running or powered-off Virtual Machines, templates, and ISOs from the target datastore to another storage location.
2. Disable vSphere HA Heartbeat: If this datastore is used for HA heartbeating, select an alternate datastore or temporarily disable HA heartbeating.
3. Disable SIOC: If Storage I/O Control (SIOC) is enabled on the datastore, disable it.
4. Remove 3rd Party Dependencies: Ensure no backup or replication tasks are actively using the datastore.

Phase 1: Unmount the Datastore

Unmounting ensures that the ESXi hosts flush pending I/O and stop accessing the VMFS file system.

1. In the vSphere Client, navigate to the Storage view.
2. Right-click the target datastore and select Unmount Datastore.
3. In the dialog box, select all ESXi hosts that access the datastore and click OK.
4. Verify the task completes successfully; the datastore status should change to Inaccessible.

Phase 2: Detach the Storage Device

Detaching prevents the host from attempting to reconnect during the maintenance, avoiding APD conditions.

1. Navigate to Hosts and Clusters and select an ESXi host.
2. Go to Configure > Storage > Storage Devices.
3. Select the iSCSI LUN associated with the array.
4. Right-click the device and select Detach.
5. Repeat this process for every ESXi host in the cluster.

Phase 3: Network Maintenance

1. Perform the subnet mask update on the storage array as per the vendor's instructions.
2. ESXi Update: On each ESXi host, navigate to Networking > VMkernel adapters and update the subnet mask for the iSCSI VMkernel (vmk) adapters to match the new network configuration.
3. Validation: Use the vmkping command from the ESXi shell to verify connectivity from the hosts to the storage array targets on the new subnet.

Phase 4: Attach and Mount the Datastore

1. In the vSphere Client, select an ESXi host and navigate to Configure > Storage > Storage Adapters.
2. Click Rescan Storage.
3. Navigate to Configure > Storage > Storage Devices.
4. Locate the detached device, right-click it, and select Attach. Repeat for all relevant hosts.
5. Navigate to the Storage view, right-click the datastore, and select Mount Datastore.