Performance issues or Slowness observed on the virtual machine

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Virtual machines experiencing performance issues running on iSCSI datastore.
• Storage array controllers used by iSCSI datastore are not load balanced or equally utilized and this is observed on the storage array.

 

• /var/run/log/vobd.log  in ESXi host is reporting Heartbeat timeout events.   

2025-01-13T05:24:35.884Z In(14) vobd[2097549]:  [vmfsCorrelator] 39837676163310us: [vob.vmfs.heartbeat.timedout] ########-####-####-####-########  [Datastore name]

2025-01-13T05:24:35.884Z In(14) vobd[2097549]:  [vmfsCorrelator] 39838001698960us: [esx.problem.vmfs.heartbeat.timedout] ########-####-####-####-########  [Datastore name]

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

This issue is caused due to a dead iSCSI adapter or the unavailable target.  

If one of the iSCSI connectivity is lost, other available paths can still be leveraged and communication with them should still continue, and there would not be any impact to the VMs or their functionality. However, there could still be a dip observed in the VMs overall performance or slowness.

iSCSI target permanently removed messages are seen from host /var/run/log/vobd.log

 

2025-01-13T05:24:24.718Z In(14) vobd[2097549]:  [iscsiCorrelator] 39837664997722us: [vob.iscsi.target.permanently.removed] The target iqn.19##-###############.a2 was permanently removed from vmhba64 login status of 0203

2025-01-13T05:24:24.718Z In(14) vobd[2097549]:  [iscsiCorrelator] 39837990533005us: [esx.problem.storage.iscsi.target.permanently.removed] The target iqn.19##-###############.a2 was permanently removed from vmhba64.

Command esxcfg-mpath -b can be used for iSCSI target adapter unavailable.

 

naa.####################: DGC iSCSI Disk (naa.#####################)

vmhba64:C1:T25:L0 LUN:0 state:dead iscsi Adapter: Unavailable Target: Unavailable

vmhba64:C0:T30:L0 LUN:0 state:active iscsi Adapter: iqn.19##-###########xi-##4:##########:64  Target: IQN=iqn.19##-###############.a3 Alias= Session=############ PortalTag=4

vmhba64:C0:T31:L0 LUN:0 state:active iscsi Adapter: iqn.19##-###########xi-##4:##########:64  Target: IQN=iqn.19##-###############.b2 Alias= Session=############ PortalTag=2

vmhba64:C0:T32:L0 LUN:0 state:active iscsi Adapter: iqn.19##-###########xi-##4:##########:64  Target: IQN=iqn.19##-###############.b3 Alias= Session=############ PortalTag=3


Resolution

Engage the storage or switch vendor to further investigate the iSCSI adapter issues which is reported to be permanently removed.

Once the dead path issues are fixed the storage controllers should be load balanced and equally utilized.

Additional Information