vSAN disk encounters medium errors but not failed out by vSAN
Article ID: 326850
Updated On:
Products
VMware vSAN
Issue/Introduction
In vmkerenl.log you see the following messages:
2022-12-24T10:46:46.161Z cpu1:2098055)ScsiDeviceIO: 4176: Cmd(0x45be86a85348) 0x28, CmdSN 0x6cc49ca2 from world 0 to dev "naa.500080d910df6c08" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 1318295484
2022-12-24T10:46:46.663Z cpu8:2098047)ScsiDeviceIO: 4176: Cmd(0x45bec4adb1c8) 0x28, CmdSN 0x6cc4a979 from world 0 to dev "naa.500080d910df6c08" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 1318296508
2022-12-24T10:46:46.679Z cpu15:2098047)HPP: HppThrottleLogForDevice:1109: Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0. from device naa.500080d910df6c08 repeated 1 times, hppAction = 1
2022-12-24T10:46:46.679Z cpu15:2098047)WARNING: HPP: HppThrottleLogForDevice:1144: Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0. hppAction = 1
2022-12-24T10:46:46.679Z cpu15:2098047)ScsiDeviceIO: 4176: Cmd(0x45be57ecd648) 0x28, CmdSN 0x6cc4a985 from world 0 to dev "naa.500080d910df6c08" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 1318295484
2022-12-24T10:46:47.180Z cpu14:2098051)ScsiDeviceIO: 4176: Cmd(0x45be6e1c4488) 0x28, CmdSN 0x6cc4b59e from world 0 to dev "naa.500080d910df6c08" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 1318296508
This can also lead to checksum errors for objects if not handled immediately
2023-01-08T05:31:52.022Z cpu27:2099333)WARNING: LSOM: LSOMReadVerifyChecksum:4397: Throttled: Checksum error detected on component 3134ba63-ded1-1658-9ed8-b49691078910, comp offset 210520969216 (computed CRC 0x0 != saved CRC 0x81bf8868 (faked: Y)
2023-01-08T05:31:56.297Z cpu19:2099333)WARNING: LSOM: LSOMScrubReadComplete:2827: Throttled: Checksum error detected on component 3134ba63-ded1-1658-9ed8-b49691078910, comp offset 210520969216 (computed CRC 0x0 != saved CRC 0x81bf8868 (faked: Y))
2023-01-08T05:32:02.042Z cpu7:2099333)WARNING: LSOM: LSOMReadVerifyChecksum:4397: Throttled: Checksum error detected on component 3134ba63-ded1-1658-9ed8-b49691078910, comp offset 210520969216 (computed CRC 0x0 != saved CRC 0x81bf8868 (faked: Y))
2023-01-05T19:03:27.576Z: [vSANCorrelator] 4820779458341us: [vob.vsan.dom.unrecoverableerror] vSAN detected an unrecoverable medium or checksum error for component 9c0cb35e-88e9-e6aa-a02c-b49691210610 on disk group 523a74b9-4262-c19d-1135-af94bf0def09.
2022-10-23T03:15:13.877Z: [vSANCorrelator] 22532796272us: [vob.vsan.dom.errorfixed] vSAN detected and fixed a medium or checksum error for component 938c5363-9c8c-d4a8-1d1b-b49691210610 on disk group 52106776-4d73-acc0-dbbe-c0bf5d36b862.
This will result in the guest operating system experiencing I/O errors, backup jobs failing, snapshots & cloning operations failing
Note: If the medium errors are detected in the metadata region vSAN will fail out the disk as per KB vSAN Disk Or Disk Group Fails With Medium Errors (81121), but when the medium errors are in the data region of the disk this doesn't happen.
2022-12-24T10:46:46.161Z cpu1:2098055)ScsiDeviceIO: 4176: Cmd(0x45be86a85348) 0x28, CmdSN 0x6cc49ca2 from world 0 to dev "naa.500080d910df6c08" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 1318295484
2022-12-24T10:46:46.663Z cpu8:2098047)ScsiDeviceIO: 4176: Cmd(0x45bec4adb1c8) 0x28, CmdSN 0x6cc4a979 from world 0 to dev "naa.500080d910df6c08" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 1318296508
2022-12-24T10:46:46.679Z cpu15:2098047)HPP: HppThrottleLogForDevice:1109: Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0. from device naa.500080d910df6c08 repeated 1 times, hppAction = 1
2022-12-24T10:46:46.679Z cpu15:2098047)WARNING: HPP: HppThrottleLogForDevice:1144: Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0. hppAction = 1
2022-12-24T10:46:46.679Z cpu15:2098047)ScsiDeviceIO: 4176: Cmd(0x45be57ecd648) 0x28, CmdSN 0x6cc4a985 from world 0 to dev "naa.500080d910df6c08" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 1318295484
2022-12-24T10:46:47.180Z cpu14:2098051)ScsiDeviceIO: 4176: Cmd(0x45be6e1c4488) 0x28, CmdSN 0x6cc4b59e from world 0 to dev "naa.500080d910df6c08" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 1318296508
This can also lead to checksum errors for objects if not handled immediately
2023-01-08T05:31:52.022Z cpu27:2099333)WARNING: LSOM: LSOMReadVerifyChecksum:4397: Throttled: Checksum error detected on component 3134ba63-ded1-1658-9ed8-b49691078910, comp offset 210520969216 (computed CRC 0x0 != saved CRC 0x81bf8868 (faked: Y)
2023-01-08T05:31:56.297Z cpu19:2099333)WARNING: LSOM: LSOMScrubReadComplete:2827: Throttled: Checksum error detected on component 3134ba63-ded1-1658-9ed8-b49691078910, comp offset 210520969216 (computed CRC 0x0 != saved CRC 0x81bf8868 (faked: Y))
2023-01-08T05:32:02.042Z cpu7:2099333)WARNING: LSOM: LSOMReadVerifyChecksum:4397: Throttled: Checksum error detected on component 3134ba63-ded1-1658-9ed8-b49691078910, comp offset 210520969216 (computed CRC 0x0 != saved CRC 0x81bf8868 (faked: Y))
2023-01-05T19:03:27.576Z: [vSANCorrelator] 4820779458341us: [vob.vsan.dom.unrecoverableerror] vSAN detected an unrecoverable medium or checksum error for component 9c0cb35e-88e9-e6aa-a02c-b49691210610 on disk group 523a74b9-4262-c19d-1135-af94bf0def09.
2022-10-23T03:15:13.877Z: [vSANCorrelator] 22532796272us: [vob.vsan.dom.errorfixed] vSAN detected and fixed a medium or checksum error for component 938c5363-9c8c-d4a8-1d1b-b49691210610 on disk group 52106776-4d73-acc0-dbbe-c0bf5d36b862.
This will result in the guest operating system experiencing I/O errors, backup jobs failing, snapshots & cloning operations failing
Note: If the medium errors are detected in the metadata region vSAN will fail out the disk as per KB vSAN Disk Or Disk Group Fails With Medium Errors (81121), but when the medium errors are in the data region of the disk this doesn't happen.
Environment
VMware vSAN 7.0.x
VMware vSAN 8.0.x
VMware vSAN 6.x
VMware vSAN 8.0.x
VMware vSAN 6.x
Resolution
To be alerted of potential medium errors in your environment and you have Log Insight create an Alert Query to send you emails. Or any 3rd party log monitoring application.
Note: There is no method to prevent the logical(physical) failure of disk blocks as disks degrade over time, therefore it's best to be proactive and have log monitoring tools monitoring the logs for these types of events and take immediate action.
Set the Scrubber to check more frequently, by default it's set to 1 for once a year, by running esxcfg-advcfg -s <value> /VSAN/ObjectScrubsPerYear
For example, you can set the value to 2 for twice a year or 4 for quarterly. VMware recommends not going any higher than 12, which would scrub once a month, to avoid any potential performance issues in the environment. If performance issues are encountered in the environment decrease the value until the optimal balance of performance/scrubbing is found for the environment.
Note: Increasing the scrubber value adds more CPU/Memory cycles which could introduce latency to the environment so don't make it too aggressive on how often the scrubber runs.
If medium errors are encountered in the data region of the disk and the disk wasn't failed out by vSAN you can do one of the following:
1) Preferred method contact the hardware vendor and get the disk replaced.
2) No-Dedup:
Remove the failed disk with medium errors from the disk group and then add it back in. Upon adding the disk back to the disk group the bad blocks will automatically be reallocated by the disk for non-use so they don't get used again
Dedup
Remove the disk group which contains the disk failing with medium errors from the host and recreate the disk group. Upon recreating the disk group the bad blocks will automatically be reallocated by the disk for non-use so they don't get used again
Note: This will result in a resync to rebuild data due to the disk/disk group being removed and then recreated/added back.
Note: There is no method to prevent the logical(physical) failure of disk blocks as disks degrade over time, therefore it's best to be proactive and have log monitoring tools monitoring the logs for these types of events and take immediate action.
Set the Scrubber to check more frequently, by default it's set to 1 for once a year, by running esxcfg-advcfg -s <value> /VSAN/ObjectScrubsPerYear
For example, you can set the value to 2 for twice a year or 4 for quarterly. VMware recommends not going any higher than 12, which would scrub once a month, to avoid any potential performance issues in the environment. If performance issues are encountered in the environment decrease the value until the optimal balance of performance/scrubbing is found for the environment.
Note: Increasing the scrubber value adds more CPU/Memory cycles which could introduce latency to the environment so don't make it too aggressive on how often the scrubber runs.
If medium errors are encountered in the data region of the disk and the disk wasn't failed out by vSAN you can do one of the following:
1) Preferred method contact the hardware vendor and get the disk replaced.
2) No-Dedup:
Remove the failed disk with medium errors from the disk group and then add it back in. Upon adding the disk back to the disk group the bad blocks will automatically be reallocated by the disk for non-use so they don't get used again
Dedup
Remove the disk group which contains the disk failing with medium errors from the host and recreate the disk group. Upon recreating the disk group the bad blocks will automatically be reallocated by the disk for non-use so they don't get used again
Note: This will result in a resync to rebuild data due to the disk/disk group being removed and then recreated/added back.
Feedback
Yes
No
Powered by
