High Read I/O on ESXi Datastores - Caused by Aggressive SNMP Polling
Article ID: 432817
Updated On:
Products
Issue/Introduction
-
Storage array reports unexplained, continuous high read throughput (e.g., 60 MBps) originating from ESXi host. The traffic is highly visible when the ESXi host is placed in Maintenance Mode (where normal virtual machine I/O is zero).
-
If several hosts exhibit the same issue, the read throughput request to the array is proportional to the number of hosts displaying similar symptoms. Thus causing excessive un-necessary load on the array.
- Performance monitoring tools such as
esxtopattribute the I/O directly to thesnmpdworld. -
The
/var/log/syslog.logfile on the affected host shows continuous, repetitive executions of theasync_collect_fsroutine:In(14) snmpd[#######]: async_collect_fs: started tid=############
less /var/log/syslog.log | grep "async_collect_fs: started"
2026-03-04T20:05:47.070Z In(14) snmpd[2099335]: async_collect_fs: started tid=330987132672
2026-03-04T20:08:46.550Z In(14) snmpd[2099335]: async_collect_fs: started tid=330987132672 =====> delta = 3 minutes
2026-03-04T20:11:47.350Z In(14) snmpd[2099335]: async_collect_fs: started tid=330987132672 =====> delta = 3 minutes
2026-03-04T22:14:47.457Z In(14) snmpd[2099335]: async_collect_fs: started tid=330987132672 =====> delta = 3 minutes
2026-03-04T22:17:48.355Z In(14) snmpd[2099335]: async_collect_fs: started tid=330987132672 =====> delta = 3 minutes
2026-03-04T22:20:47.683Z In(14) snmpd[2099335]: async_collect_fs: started tid=330987132672 =====> delta = 3 minutes
Environment
VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 9.0.x
Cause
This behavior is caused by an external Network Management System (NMS) aggressively polling the ESXi host for storage metrics via SNMP (UDP port 161).
When an NMS requests the HOST-RESOURCES-MIB, specifically the hrStorageTable OIDs (such as .1.3.6.1.2.1.25.2.3.1.5 for hrStorageSize (total capacity) and .1.3.6.1.2.1.25.2.3.1.6 for hrStorageUsed (consumed capacity))—the ESXi snmpd daemon must provide real-time datastore capacity and allocation data. To fulfill the request, snmpd invokes the async_collect_fs internal worker routine.
async_collect_fs issues active SCSI commands (such as Read Capacity and VMFS metadata reads) directly down the Fibre Channel storage stack to every presented LUN. If the NMS polling interval is exceptionally aggressive (e.g., every 1 to 3 minutes), it generates a continuous stream of metadata reads that materializes as high SAN throughput.
Resolution
Identify the world/service generating the IO to the array at ESXi level
Since the host is in maintenance, virtual machine I/O is zero. Generate esxtop batch data and utilize windows performance monitor to isolate the world ID generating the load.
Generate esxtop batch data for 10 minutes from an ssh session to the host by running this command below after keying in a valid datastore name:
# minutes=10; path="/vmfs/volumes/DATASTORE_WITH_ENOUGH_SPACE"; esxtop -ba -d 2 -n $(expr ${minutes} \* 30) > "${path}"/$(hostname)_$(date -u +"%Y-%m-%dT%H%M%S")_esxtop_batch_all.csv
Load the CSV file to performance monitor tool from windows:
Run: perfmon
In the left-hand pane, click on Performance Monitor under "Monitoring Tools".
Right click on the graph and select “Properties”.
Select the “Source” tab.
Select the “Log files:” radio button from the “Data source” section.
Click the “Add” button.
Select the CSV file created by esxtop and click “OK”.
Click the “Apply” button.
Select Counters to Display:
Click the Data tab.
Click Add >> Under 'Physical Disk Per-Device-Per-Wrold', select 'MBytes Read/sec' and 'Reads/sec' (or one at a time) >> click Add >> click OK.
Visualize the Data:
The graph will now populate, change the view from "Line" to "Histogram" or "Report" using the toolbar icons or Ctrl + G.
Hover the mouse over the graph to identify world ID leading to a high throughput. Worlds with negligible throughputs can be ignored.
With the world ID, run this command to identify the service associated with the world
# ps -c | grep <world-id>
example: # ps -c | grep 2099335
2099335 2099335 snmpd /sbin/snmpd
Catch the poller in the act with a packet capture
snmpd is merely the messenger, the only way to identify the true source generating the load is to look at the network origin of the requests feeding snmpd which operates on UDP port 161.
From an SSH session run this command
# tcpdump-uw -i vmk0 -n -nn udp port 161
Note: Replace vmk0 if your management network resides on a different vmkernel port.
[root@esxi-host:~] tcpdump-uw -i vmk0 -n -nn udp port 161
tcpdump-uw: verbose output suppressed, use -v[v]... for full protocol decode
listening on vmk0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:50:30.029669 IP ###.###.32.71.59841 > ###.###.3.30.161: C="********" GetRequest(30) .1.3.6.1.2.1.25.2.3.1.3.10
21:50:30.030034 IP ###.###.3.30.161 > ###.###.32.71.59841: C="********" GetResponse(79) .1.3.6.1.2.1.25.2.3.1.3.10="/vmfs/volumes/60595029-21ee8223-828e-0025b5070117"
21:50:30.030376 IP ###.###.32.71.59841 > ###.###.3.30.161: C="********" GetRequest(30) .1.3.6.1.2.1.25.2.3.1.3.11
21:50:30.030430 IP ###.###.3.30.161 > ###.###.32.71.59841: C="********" GetResponse(79) .1.3.6.1.2.1.25.2.3.1.3.11="/vmfs/volumes/605966b4-cc470988-d7c6-0025b5070003"
21:50:30.030738 IP ###.###.32.71.59841 > ###.###.3.30.161: C="********" GetRequest(30) .1.3.6.1.2.1.25.2.3.1.3.13
21:50:30.030782 IP ###.###.3.30.161 > ###.###.32.71.59841: C="********" GetResponse(79) .1.3.6.1.2.1.25.2.3.1.3.13="/vmfs/volumes/605b4f8d-7519b8f6-4fe6-0025b5070003"
21:50:30.031027 IP ###.###.32.71.59841 > ###.###.3.30.161: C="********" GetRequest(30) .1.3.6.1.2.1.25.2.3.1.3.16
21:50:30.031072 IP ###.###.3.30.161 > ###.###.32.71.59841: C="********" GetResponse(79) .1.3.6.1.2.1.25.2.3.1.3.16="/vmfs/volumes/62baabb9-76ea5e28-e9b4-0025b522026e"
Where .1.3.6.1.2.1.25.2.3.1.3 is a request for names of the mount points/datastores and ###.###.32.71 is the external NMS.
Resolution:
To permanently resolve the excessive storage I/O, change the monitoring architecture by migrating to vCenter API Monitoring.
Shift the storage metrics collection from localized ESXi SNMP polling to centralized vCenter Server API polling.
Provision a read-only service account in vCenter Single Sign-On.
Do not use administrator credentials for the monitoring tool. Create a dedicated service account with the Principle of Least Privilege.
Log into the vSphere Client as an SSO administrator.
Navigate to Administration > Single Sign-On > Users and Groups.
Create a new user (e.g., [email protected]).
Navigate to Menu > Global Permissions.
Add [email protected] and assign the Read-only role. Ensure the Propagate to children checkbox is selected so the account can read datastore capacity across all clusters.
Reconfigure the external NMS to target the vCenter Server API for datastore capacity and utilization metrics. vCenter caches these metrics natively, eliminating the need for host-level SCSI interrogation.
Log into your external monitoring tool.
Add a new device/node targeting the vCenter Server FQDN or IP.
Select the monitoring type corresponding to VMware APIs (often labeled "VMware/vCenter API", "SOAP/REST", or "vSphere Sensor" depending on the vendor).
Input the [email protected] credentials.
Remove the ESXi hosts from the NMS SNMP polling inventory.
Workaround: Adjust NMS Polling Intervals
If API migration is not immediately feasible, reconfigure the external NMS to reduce the polling frequency.
Increase the SNMP polling interval for storage metrics to a minimum of 15 to 30 minutes.
Alternatively, restrict the NMS SNMP view to exclude the HOST-RESOURCES-MIB entirely if storage metrics are not required.
Feedback
