A vSAN health test alarm 'Physical disk health retrieval issues' is triggered and immediately resolved

Products

• VMware vSAN 8.x

Issue/Introduction

• An alarm 'Physical disk health retrieval issues' is triggered and immediately resolved.
  
  MMM DD HH:25:24 <vCenter Server Name> vpxd[1508173]: Event [35063280] [1-1] [YYYY-MM-DDTHH:25:24.026592Z] [vim.event.EventEx] [error] [vSAN Health] [<Data Center Name>] [35063280] [vSAN Health Test 'Physical disk health retrieval issues' status changed from 'green' to 'red']

MMM DD HH:27:38 <vCenter Server Name> vpxd[1508173]: Event [35063304] [1-1] [YYYY-MM-DDTHH:27:38.745332Z] [vim.event.EventEx] [info] [vSAN Health] [<Data Center Name>] [35063304] [vSAN Health Test 'Physical disk health retrieval issues' status changed from 'red' to 'green']


• /var/log/vmware/vsan-healath/vmware-vsan-health-service log shows messages similar to:
  
  YYYY-MM-DDTHH:25:08.829Z ERROR vsan-mgmt[1530866] [VsanClusterHealthSystemImpl::PerHostQueryHealthThreadMain opID=noOpId] method fault for host <ESXi Host Name>
Traceback (most recent call last):
  File "bora/vsan/health/esx/pyMo/VsanClusterHealthSystemImpl.py", line 2620, in PerHostQueryHealthThreadMain
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 598, in <lambda>
  self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 388, in _InvokeMethod
  return self._stub.InvokeMethod(self, info, args)
File "bora/vsan/health/esx/pyMo/VsanClusterHealthSystemImpl.py", line 414, in InvokeMethod
File "bora/vsan/health/vpxd/pyMoVsan/VsanHealthHelpers.py", line 4161, in GetResultFromHostInfo
PyCppVmomi.vim.fault.VsanFault: (vim.fault.VsanFault) {
faultMessage = (vmodl.LocalizableMessage) [
  (vmodl.LocalizableMessage) {
    key = 'FindEntry failed: Not found, query.uuid=<Disk UUID>, subClusterUuid=<vSAN Sub-Cluster UUID>,
    message = 'FindEntry failed: Not found, query.uuid=<Disk UUID>, subClusterUuid=<vSAN Sub-Cluster UUID>'
  }
]
}

YYYY-MM-DDTHH:25:24.012Z INFO vsan-mgmt[1513490] [VsanEventUtil::_generateVcEvent opID=noOpId] Generate VC event for managed object <Cluster Name> with testName=com.vmware.vsan.health.test.physicalDisks.physdiskHostIssues.testName, testId=com.vmware.vsan.health.test.physdiskhostissues, preStatus=green, curStatus=red
YYYY-MM-DDTHH:25:24.013Z INFO vsan-mgmt[1513490] [VsanEventUtil::_generateVcEvent opID=noOpId] Post event = (vim.event.EventEx) {
key = 0,
chainId = 0,
createdTime = YYYY-MM-DDT05:25:24.013027Z,
userName = 'vSAN Health',
computeResource = (vim.event.ComputeResourceEventArgument) {
  name = '<Cluster name>',
  computeResource = 'vim.ClusterComputeResource:domain-c####'
},
eventTypeId = 'vsan.health.test.physicaldisks.physdiskhostissues.event',
severity = 'error',
arguments = (vmodl.KeyAnyValue) [
  (vmodl.KeyAnyValue) {
    key = 'prestatus',
    value = 'green'
  },
  (vmodl.KeyAnyValue) {
    key = 'curstatus',
    value = 'red'
  }
]
}

YYYY-MM-DDTHH:27:38.743Z INFO vsan-mgmt[1513490] [VsanEventUtil::_generateVcEvent opID=noOpId] Generate VC event for managed object <Cluster Name> with testName=com.vmware.vsan.health.test.physicalDisks.physdiskHostIssues.testName, testId=com.vmware.vsan.health.test.physdiskhostissues, preStatus=red, curStatus=green
YYYY-MM-DDTHH:27:38.743Z INFO vsan-mgmt[1513490] [VsanEventUtil::_generateVcEvent opID=noOpId] Post event = (vim.event.EventEx) {
  key = 0,
  chainId = 0,
createdTime = YYYY-MM-DDTHH:27:38.743484Z,
  userName = 'vSAN Health',
  computeResource = (vim.event.ComputeResourceEventArgument) {
  name = '<Cluster Name>',
  computeResource = 'vim.ClusterComputeResource:domain-c####'
  },
  eventTypeId = 'vsan.health.test.physicaldisks.physdiskhostissues.event',
  severity = 'info',
  arguments = (vmodl.KeyAnyValue) [
    (vmodl.KeyAnyValue) {
      key = 'prestatus',
      value = 'red'
    },
    (vmodl.KeyAnyValue) {
      key = 'curstatus',
      value = 'green'
    }
  ]
}



• No other vSAN health alarms triggered.
  
  
• No errors are seen on the ESXi host.

Environment

VMware vSAN 8.0

Cause

The disk health status may have been updated during disk read operations, which could be the cause of the health retrieval failure.

Resolution

vSAN 9.0 addresses this issue.