VCF Administrator Noting vRealize Operation is Reporting RX on Multiple Hosts.

Products

• VMware vSphere ESXi
• VMware vSAN 8.x

Issue/Introduction

vSAN Host experiencing performance and re-sync issues.

Environment

• vSAN ESA
• vSAN OSA
• ESXi All Supported Versions

Cause

The issue foot print below is often indicative of a hardware./configuration issue at the controller or device layer.

Reported in vmkernel.log, earlier messages indicate diskgroup device reporting as missing, or unable to be Read from

• vmkwarning: )WARNING: PLOG: PLOGProbeDevice:6851: Failed to read the device <Device ID> 0:8> : Not found
• vmkwarning: )WARNING: PLOG: PLOGProbeDevice:6851: Failed to read the device <Device ID> 0:7> : Not found 

 

• WARNING: PLOG: PLOGProbeDevice:####: Failed to read the device <Device ID>: Not found

 

• vmkwarning: cpu41:####)WARNING: LinuxFileDesc: ###: python3: Unrecoverable exec failure: Failure during exec while original state already lost
•  vmkwarning: cpu71:####)WARNING: Vol3: 4678: Error closing the volume: . Eviction fails: Failure

Resolution

When this type of error profile is observed in a vSAN environment it is recommended to immediately engage the hardware vendor to perform a health-check/hardware review.

Broadcom Vmware GS can assist in identifying the issue, but true root cause/ultimate resolution will need to come form the vendor

Additional Information

The above messages further illustrate vSAN/ESXi's inability to manage access the device in question, even to the point where vSAN is unable to evict the disk from the disk-group in a 'return to service effort'

Either the Cache Disk or the Capacity Tier disk or both could have been affected in an event similar to the above.