Facing an I/O error on a VM which is hosted on VMware Environment.
search cancel

Facing an I/O error on a VM which is hosted on VMware Environment.

book

Article ID: 377677

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • In the VM logs - /var/log/messages - you may see messages similar to the ones below:
  • YYYY-MM-DDTHH:MM:SS VM-NAME kernel: JBD2: Detected IO errors while flushing file data on dm-2-8
YYYY-MM-DDTHH:MM:SS VM-NAME kernel: JBD2: Detected IO errors while flushing file data on dm-2-8
YYYY-MM-DDTHH:MM:SS VM-NAME kernel: JBD2: Detected IO errors while flushing file data on dm-2-8
  • In the /var/log/vmkernel.log on the ESXi host where the impacted VM resides, you may see errors related to the datastore where the VM's files reside, as the sample:
  • YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu52:136841550)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu60:147041128)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu48:147071119)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu82:147041126)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu3:147041137)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu79:147041125)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu55:146964280)Fil6: 4179: 'DATASTORE-NAME': Fil6 file IO (<FD c60 r110>) : Limit exceeded
  • The datastore on the messages above was resized prior to the start of the errors/problems.
  • If the datastore LUNs are thin provisioned and the backing storage pool capacity is exhausted. VM IOs may fails with same error even after freeing space on the datastore. 

Environment

  • VMware vSphere 7.x
  • VMware vSphere 8.x

Cause

The datastore where the VM's files reside was expanded, but not all ESXi hosts that are connected to it were refreshed with the new information and are not aware of the new size of the datastore. So, the VM tries to send IO operations to a block that the host believes is outside the datastore, causing the issues seen. When a storage rescan is not run, or if it fails on a host, then, when that host next queries the volume metadata it will find a mismatch between the extended volume size recorded and its cached record of the original smaller device size.

Resolution

Perform a storage rescan on all hosts that are connected to this datastore, to have the new size information refreshed in all of them.

Additional Information

Powered by Wolken Software