sudden increase in data rate on switch ports.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms : 

• High data rate detected on switch ports connected to particular host.
  

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

• Number of active paths configured on affected host are less compared to other hosts in the cluster.
• VMs on affected host generating high IOs.

Cause validation:

• Verify the ESXTOP data to validate if any VMs generating high IOs.
  
  
  Refer Using esxtop to identify storage performance issues for ESXi (multiple versions) for more information.
  
  
• Run the command "esxcfg-mpath -b -d <naa.ID>" on affected host.
  
  esxcfg-mpath -b -d naa.################################
naa.################################
   Device Display Name: IBM Fibre Channel Disk (naa.################################)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=17,TPG_state=ANO}{TPG_id=16,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba2:C0:T4:L47, vmhba65:C0:T3:L47
   Is USB: false
  
  From the above output it is confirmed that only 2 active paths with round robin policy (iops=1000) are configured for datastores on affected host.
  
  
• Run the command "esxcfg-mpath -b -d <naa.ID>" on healthy host.
  
  esxcfg-mpath -b -d naa.################################
naa.################################
   Device Display Name: IBM Fibre Channel Disk (naa.################################)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=17,TPG_state=ANO}{TPG_id=16,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba3:C0:T0:L47, vmhba65:C0:T7:L47, vmhba64:C0:T1:L47, vmhba2:C0:T2:L47
     Is USB: false
  
  From the above output it is confirmed that 4 active paths with round robin policy (iops=1000) are configured for datastores on healthy host.

Resolution

• Migrate the VM generating high I/O to a healthy host configured with four active paths to the datastores, in order to reduce the load on the affected host.
• Engage the application team to investigate and resolve the high I/O generation on the VM.
• Configure four active paths on the affected host to ensure consistent and uniform storage path configuration across all hosts in the cluster.