How to simulate Fibre Channel link failure for Epic validation on ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Administrators managing Epic healthcare environments on VMware vSphere often require a method to validate High Availability (HA) and storage multipathing configurations. This requires simulating a "link down" or "dead path" scenario for Fibre Channel (FC) Host Bus Adapters (HBAs) to ensure traffic fails over correctly to redundant paths.

Environment

 

• VMware vSphere ESXi 8.0 u3

• Fibre Channel (FC) Storage

Resolution

Perform one of the following methods to simulate the failure.

Method 1: Logical Simulation (Path Isolation) This method effectively mimics a "link down" state to the storage layer without physically disconnecting cables.

1. Connect to the ESXi host via SSH as root.

2. Identify the target HBA and active paths:

   esxcli storage san fc list
   esxcli storage core path list| grep -i <vmhba_adapter>
   

   Note the "Runtime Name" (e.g., vmhba1:C0:T0:L1) or "Path UID".

3. Disable the specific path to cut I/O:

   esxcli storage core path set --state off --path <Runtime_Name_or_UID>
   

4. Verify the path state is "off":

   esxcli storage core path list | grep -A 10 <vmhba_adapter>
   

5. Restore the path after testing:

   esxcli storage core path set --state active --path <Runtime_Name_or_UID>
   

Method 2: Physical Simulation This triggers actual hardware-level link-down events.

The following is just a reference for actual steps; please reach out to your physical fiber switch vendor for switch port shutdown.

1. Log in to the FC Switch (Brocade/Cisco MDS).

2. Identify the port connected to the ESXi host's HBA (using WWPN from esxcli storage san fc list).

3. Disable the port:

   • Cisco MDS: switch# config t, switch(config)# interface fc1/1, switch(config-if)# shutdown

   • Brocade: switch:admin> portdisable <port_number>

4. Verify on ESXi:

   esxcli storage san fc list

   Status should change to LINK DOWN.

5. Restore: Enable the port (no shutdown / portenable).

Method 3: Link Reset (Driver Reset) To test driver recovery mechanisms (LIP/FLOGI reset) without a permanent state change:

1. Execute the reset command against the adapter:

   esxcli storage san fc reset -A <vmhba_adapter>

Additional Information

 

• Modifying path information for ESXi hosts

• Forcing a fabric login reset on Fibre Channel HBAs

If disabling the specific path to cut I/O fails:
e.g.

 esxcli storage core path set --state off  --path vmhba1:C0:T0:L0
Unable to set path state. Error was: Unable to change path state, the path is marked as 'busy' by the VMkernel.: Busy: VSI node (1966:VSI_NODE_storage_scsifw_paths_state)

This error indicates the VMkernel is actively using this path for critical I/O (e.g., active VM traffic, VMFS heartbeating, or it is a Boot-from-SAN device) and refuses to gracefully disable it.

Since the kernel is blocking the logical simulation, you must proceed with the physical simulation to force the failover event.