Unexpected network traffic is observed between two ESXi hosts
search cancel

Unexpected network traffic is observed between two ESXi hosts

book

Article ID: 438152

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • A higher than expected amount of network traffic is observed between two ESXi hosts over their management vmkernel interfaces
    • This traffic occurs for a relatively short period of time and then ceases
  • Entries similar to the following are seen in the /var/run/log/vmkernel.log file on the destination ESXi host:

    2026-04-15T15:26:16.730Z In(166) Vpxa[2101948]: [Originator@6876 sub=vpxLro opID=3c254077-15] [VpxLRO] -- BEGIN task-92819 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange -- 52223a2a-####-####-####-f8347c05191a
    2026-04-15T15:26:16.734Z In(166) Vpxa[2103473]: [Originator@6876 sub=vpxLro opID=3c254077-15] [VpxLRO] -- FINISH task-92819
    2026-04-15T15:26:23.956Z In(166) Vpxa[2110607]: [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-5e29b5f7-e6] [VpxLRO] -- BEGIN lro-533815 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 52223a2a-####-####-####-f8347c05191a
    2026-04-15T15:26:23.956Z In(166) Vpxa[2110607]: [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-5e29b5f7-e6] [VpxLRO] -- FINISH lro-533815
    2026-04-15T15:26:30.460Z In(166) Vpxa[2101902]: [Originator@6876 sub=vpxLro opID=mf5kevrc-83167749-auto-1diknf-h5:76424369-11-6e] [VpxLRO] -- BEGIN lro-533816 -- vpxa -- vpxapi.VpxaService.searchDatastore -- 52223a2a-4faa-2ef8-4963-f8347c05191a
    2026-04-15T15:26:30.460Z In(166) Vpxa[2101902]: [Originator@6876 sub=vpxaMoService opID=mf5kevrc-83167749-auto-1diknf-h5:76424369-11-6e] SearchDatastore: searching osfs datastore
    2026-04-15T15:26:30.470Z In(166) Vpxa[2101902]: [Originator@6876 sub=vpxLro opID=mf5kevrc-83167749-auto-1diknf-h5:76424369-11-6e] [VpxLRO] -- FINISH lro-533816
    ...
    2026-04-15T15:26:50.964Z In(166) Vpxa[2104129]: [Originator@6876 sub=vpxLro opID=mf5kevrc-83167938-auto-1diksk-h5:76424380-48-3f] [VpxLRO] -- BEGIN task-92820 -- vpxa -- vpxapi.VpxaService.nfcCopy -- 52223a2a-####-####-###-f8347c05191a
    2026-04-15T15:26:50.964Z In(166) Vpxa[2104129]: [Originator@6876 sub=hostdds opID=mf5kevrc-83167938-auto-1diksk-h5:76424380-48-3f] Root directory: '/vmfs/volumes/vsan:################-################/', display name: 'a4e9c964-####-####-####-f4c7aa50157c/26100.32230.260111-0550.<FILE NAME>'
    ...
    2026-04-15T15:49:39.282Z In(166) Vpxa[2104129]: [Originator@6876 sub=Libs opID=mf5kevrc-83167938-auto-1diksk-h5:76424380-48-3f] [NFC INFO]NfcSessionStats: session=4042DDE4F0, type=client, clientName='VMware ESXi', streamMode=1, fssrvrMode=0, aioMode=0, version=11, remoteVersion=11, currState=NFC_IDLE, prevState=NFC_IDLE, returnCode=NFC_SUCCESS (0), detail="", opID='vpxapi.VpxaService.nfcCopy', sessionDurationUs=1367332708, sessionIdleTimeUs=1367332679 (100.0%), fileIoIdleTimeUs=1325585676 (96.9%), attemptedFileTransfers=1, successfulFileTransfers=1, totalBytesTransferred=8152356864, totalFilesSize=0, fileLockFailureCount=0
    2026-04-15T15:49:39.282Z In(166) Vpxa[2104129]: [Originator@6876 sub=Libs opID=mf5kevrc-83167938-auto-1diksk-h5:76424380-48-3f] [NFC INFO]readIoLatencyStats: count 31100 min/max/avg 8/57901/1341 usec
    2026-04-15T15:49:39.282Z In(166) Vpxa[2104129]: [Originator@6876 sub=Libs opID=mf5kevrc-83167938-auto-1diksk-h5:76424380-48-3f] [NFC INFO]netSendLatencyStats: count 62214 min/max/avg 0/4549292/21250 usec
    2026-04-15T15:49:39.282Z In(166) Vpxa[2104129]: [Originator@6876 sub=Libs opID=mf5kevrc-83167938-auto-1diksk-h5:76424380-48-3f] [NFC INFO]netRecvLatencyStats: count 7 min/max/avg 7/1614967/399309 usec
    2026-04-15T15:49:39.282Z In(166) Vpxa[2104129]: [Originator@6876 sub=vpxLro opID=mf5kevrc-83167938-auto-1diksk-h5:76424380-48-3f] [VpxLRO] -- FINISH task-92820

Environment

VMware vSphere ESXi

Cause

The file name noted in the /var/run/log/vmkernel.log snippet above by <FILE NAME> was being copied from the source ESXi host to the destination ESXi host as a part of a catalog synchronization and possibly a VM deployment. This data transfer is an nfc file copy and happens over the ESX hosts' management network by default.

Resolution

This is expected behavior.

Additional Information

See Configuring Dedicated vSphere Provisioning Service for Cold Migration Network Traffic  to configure a provisioning interface to avoid this type of traffic being sent over the management interface.

Powered by Wolken Software