vmfs uuid change cause "Corrupt heartbeat detected" and "hostd detected to be non-responsive" error

Products

• VMware vSphere ESXi

Issue/Introduction

Below error recorded in vmkernal log:

2020-08-20T03:56:02.343Z cpu27:65678)WARNING: HBX: 727: 'Datastore###': HB at offset 12183761018402815236 - Volume ######-######-####-88df9e365330 may be damaged on disk. Corrupt heartbeat detected:
2020-08-20T03:56:02.349Z cpu44:65679)WARNING: HBX: 727: 'Datastore###': HB at offset 12183761018402815236 - Volume ######-######-####-88df9e365330 may be damaged on disk. Corrupt heartbeat detected:
2020-08-20T03:56:02.350Z cpu28:65680)WARNING: HBX: 727: 'Datastore###': HB at offset 12183761018402815236 - Volume ######-######-####-88df9e365330 may be damaged on disk. Corrupt heartbeat detected:
2020-08-20T03:56:02.353Z cpu27:65678)WARNING: HBX: 727: 'Datastore###': HB at offset 12183761018402815236 - Volume ######-######-####-88df9e365330 may be damaged on disk. Corrupt heartbeat detected:
2020-08-20T03:56:02.356Z cpu44:65679)WARNING: HBX: 727: 'Datastore###': HB at offset 12183761018402815236 - Volume ######-######-####-88df9e365330 may be damaged on disk. Corrupt heartbeat detected:
...
2020-08-20T06:05:09.848Z cpu43:65680)WARNING: HBX: 727: 'Datastore###': HB at offset 12183761018402815236 - Volume ######-######-####-88df9e365330 may be damaged on disk. Corrupt heartbeat detected:
2020-08-20T06:05:12.155Z cpu51:65679)WARNING: HBX: 727: 'Datastore###': HB at offset 12183761018402815236 - Volume ######-######-####-88df9e365330 may be damaged on disk. Corrupt heartbeat detected:
2020-08-20T06:05:15.155Z cpu27:65678)WARNING: HBX: 727: 'Datastore###': HB at offset 12183761018402815236 - Volume ######-######-####-88df9e365330 may be damaged on disk. Corrupt heartbeat detected:

Found host non-response due to the datastore connection issue:

2020-08-20T04:30:01.289Z cpu30:7593544)ALERT: hostd detected to be non-responsive
2020-08-20T05:00:01.557Z cpu53:7593789)ALERT: hostd detected to be non-responsive
2020-08-20T05:30:01.826Z cpu35:7593863)ALERT: hostd detected to be non-responsive

Cannot find the target vmfs uuid in customer site but the same naa.id point to another datastore path

Environment

VMware vSphere ESXI 7

VMware vSPhere ESXI 8

Cause

ESXi was installed on a shared datastore instead of a local disk. The VMFS UUID of the shared storage was changed, therefore the VMFS corruption error message is recorded.

Resolution

Installing ESXi in a shared location will overwrite the data in the storage location. This is expected behavior.

Ensure a proper local storage location is used when installing ESXI.