Facing an I/O error on a VM which is hosted on VMware Environment.
search cancel

Facing an I/O error on a VM which is hosted on VMware Environment.

book

Article ID: 377677

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • In the VM logs - /var/log/messages - you may see messages similar to the ones below:
Sep 12 11:28:47 VM-NAME kernel: JBD2: Detected IO errors while flushing file data on dm-2-8
Sep 12 11:28:49 VM-NAME kernel: JBD2: Detected IO errors while flushing file data on dm-2-8
Sep 12 11:28:52 VM-NAME kernel: JBD2: Detected IO errors while flushing file data on dm-2-8
  • In the vmkernel logs of the ESXi host where the impacted VM resides, you may see errors related to the datastore where the VM's files reside, as the sample:
2024-09-12T08:28:45.745Z In(182) vmkernel: cpu52:136841550)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
2024-09-12T08:28:45.746Z In(182) vmkernel: cpu60:147041128)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
2024-09-12T08:28:45.747Z In(182) vmkernel: cpu48:147071119)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
2024-09-12T08:28:45.747Z In(182) vmkernel: cpu82:147041126)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
2024-09-12T08:28:45.750Z In(182) vmkernel: cpu3:147041137)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
2024-09-12T08:28:45.751Z In(182) vmkernel: cpu79:147041125)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
2024-09-12T08:28:45.752Z In(182) vmkernel: cpu55:146964280)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
  • The datastore on the messages above was resized prior to the start of the errors/problems.

Environment

VMware vSphere 7.0.x

VMware vSphere 8.0.x

Cause

The datastore where the VM's files reside was expanded, but not all ESXi hosts that are connected to it were refreshed with the new information and are not aware of the new size of the datastore. So the VM tries to send IO operations to a block that the host believes is outside the datastore, causing the issues seen.

In other words, if a storge rescan is not run, or if it fails on a host, then, when that host next queries the volume metadata it will find a mismatch between the extended volume size recorded and its cached record of the original smaller device size.

Resolution

Perform a storage rescan on all hosts that are connected to this datastore, to have the new size information refreshed in all of them.

Additional Information

 

Powered by Wolken Software