Renaming VMFS datastores in bulk causes high read I/O spikes

Products

• VMware vSphere ESXi

Issue/Introduction

• Multiple VMFS datastores are renamed in quick succession, for example via an automation script (e.g., PowerCLI)
• Read I/O spikes are detected at the storage layer during datastore rename tasks.
• Storage array front-end ports may become saturated.
• Potential performance degradation or outages on systems sharing the same storage resources.

Environment

VMware vSphere ESXi 8.x
VMFS Datastores

Cause

Renaming a VMFS datastore triggers a background metadata refresh across the environment. This process executes SlowRefresh operations on the ESXi hosts, which generate a spike in read I/O to process volume metadata.

Performing these renames in bulk leads to concurrent metadata reads from multiple hosts, which can overwhelm the storage array controllers.

While the datastore name change may appear to complete in a short period of time, the background metadata reads start after the rename task has completed.

####-##-##T##:##:##.###Z In(166) Hostd[2099719] [Originator@6876 sub=Vimsvc.TaskManager [email protected]:323-7bacb338-4c84 sid=5233feb6 user=vpxuser] Task Created : haTask-########-########-####-############-vim.Datastore.renameDatastore-############
####-##-##T##:##:##.###Z In(166) Hostd[2099716] [Originator@6876 sub=Vimsvc.TaskManager [email protected]:323-7bacb338-4c84 sid=5233feb6 user=vpxuser] Task Completed : haTask-########-########-####-############-vim.Datastore.renameDatastore-############ Status success
####-##-##T##:##:##.###Z In(166) Hostd[2099705] [Originator@6876 sub=Vimsvc.TaskManager [email protected]:323-7bacb338-4c8a sid=5233feb6 user=vpxuser] Task Created : haTask-ha-host-vim.host.StorageSystem.refresh-1947215912
####-##-##T##:##:##.###Z In(166) Hostd[2099706] [Originator@6876 sub=Libs] SlowRefresh: path /vmfs/volumes/########-########-####-############ total blocks ############## used blocks ##############forceRefresh = 0
####-##-##T##:##:##.###Z In(166) Hostd[2099718] [Originator@6876 sub=Libs] SlowRefresh: path /vmfs/volumes/########-########-####-############ total blocks ############## used blocks ##############forceRefresh = 0
####-##-##T##:##:##.###Z In(166) Hostd[2099728] [Originator@6876 sub=Libs] SlowRefresh: path /vmfs/volumes/########-########-####-############ total blocks ############## used blocks ##############forceRefresh = 0

 

 

Resolution

To prevent storage overload during datastore renames:

1. Initiate the rename operation for a single datastore.
2. Monitor the read I/O on the storage array.
3. Wait for the read I/O to return to its normal baseline level before initiating the next rename task.