High Single-Core CPU Utilization During Large Data Transfers Due to Single Traffic Flow

Products

• VMware NSX

Issue/Introduction

• In certain scenarios, observe high utilization (up to 100%) on a single CPU core while performing large data transfer operations, such as backups or replications. Other CPU cores may remain underutilized. This behavior occurs when the traffic is handled as a single flow from application side.

Sample:

edge> get dataplane cpu stats


CPU Usage
Core      : 0
Crypto    : 0 pps
Intercore : 0 pps
Kni       : 0 pps
Rx        : 14621 pps
Slowpath  : 20 pps
Tx        : 15868 pps
Usage     : 10%

Core      : 1
Crypto    : 0 pps
Intercore : 0 pps
Kni       : 0 pps
Rx        : 4883 pps
Slowpath  : 0 pps
Tx        : 5791 pps
Usage     : 6%

Core      : 2
Crypto    : 0 pps
Intercore : 0 pps
Kni       : 0 pps
Rx        : 264427 pps
Slowpath  : 0 pps
Tx        : 265383 pps
Usage     : 99% <<<<<<


Core      : 3
Crypto    : 0 pps
Intercore : 0 pps
Kni       : 0 pps
Rx        : 9160 pps
Slowpath  : 0 pps
Tx        : 9912 pps
Usage     : 7%

>>>>>

• When a large data transfer or backup job runs, the system may show one CPU core fully utilized while other cores have available capacity.
• Packet captures when taken on the Edge uplink interface, it will reveal that all traffic associated with the transfer belongs to a single flow sharing the same five-tuple values (Source IP, Destination IP, Source Port, Destination Port, and Protocol).

 

Environment

VMware NSX

Cause

• The traffic pattern consists of a single flow with identical five-tuple parameters. 
• When the backup job runs on the application, the traffic is directed to a single flow handled by one CPU core
• NSX computes a hash based on the five-tuple hashing for load distribution across CPU cores.
• When only one flow exists, there is no opportunity for load balancing between cores.

 

Resolution

• This is expected product behavior and not a defect.

Recommendation Actions: 

• Collect NSX logs and ADF data during the high CPU utilization event.
• Verify that the data transfer if it uses a single flow (same source/destination IP and port, and same protocol).
• Configure the data transfer or backup application to use multiple streams or flows.
• Each stream generates a unique five-tuple, allowing NSX to distribute traffic across multiple CPU cores.
• Most modern backup or replication tools include options to enable multi-stream or multi-threaded transfers.

Example:
Backup software such as Veeam Backup & Replication supports multithreaded data transfer, which can be enabled to distribute workload efficiently.
Refer to vendor documentation for enabling multi-stream or parallel data transfer options- https://helpcenter.veeam.com/docs/backup/vsphere/enabling_multithreaded_transfer.html?ver=120

 

 

Additional Information

• Adjusting NSX-level parameters (e.g., NetIOC tuning, pNIC ring size, flow cache, or action affinity) will not resolve the single-flow utilization issue.
• The optimal resolution lies in enabling multiple flows from the application side.