VM Edge Node "kni_single" Core CPU Usage Very High Despite Low Network Traffic

Products

• VMware NSX

Issue/Introduction

•  The 'kni_single' thread's CPU usage reaches over 95% to 100% in a VM Edge environment:

 

• The usage does not decrease, even after network traffic and dataplane CPU core usage drop:

 

• When checking the stats for KNI Interfaces, the interface "span-0" is present, and its drop counter is high.

edge-appctl -t /var/run/vmware/edge/dpd.ctl kni/show | json_pp
      "dev_name" : "span-0", --------------------------!!!
      "ifuid" : 390,
      "ifuuid" : "###-###-###-###-###",
      "lcore" : 0,
      "mac" : "##:##:##:##:##:##",
      "mbuf_alloc_errors" : 0,
      "stats" : {
         "rx_bytes" : 0,
         "rx_drops" : 0,
         "rx_pkts" : 0,
         "tx_bytes" : 61327396472069,
         "tx_drops" : 35654560,    ----------------------!!!
         "tx_pkts" : 93852422480
      },
      "type" : 0
 
      "dev_name" : "kni-lrport-0",
      "ifuid" : 259,
      "ifuuid" : "###-###-###-###-###",
      "lcore" : 0,
      "mac" : "##:##:##:##:##:##",
      "mbuf_alloc_errors" : 0,
      "stats" : {
         "rx_bytes" : 196906306406,
         "rx_drops" : 0,
         "rx_pkts" : 2745172567,
         "tx_bytes" : 143400350519,
         "tx_drops" : 0,
         "tx_pkts" : 1570210353
      },
      "type" : 0

 

Environment

VMware NSX 4.1.0

Cause

L7 LB packets are forwarded to the TCP/IP stack via the KNI sub-interface, so in most cases, the "kni_single" thread is utilized when L7 virtual servers experience high network traffic.

However, in this scenario, the "span-#" interface is created when the command "set capture session #" is executed to capture packets on the edge node.

Even if your environment only uses L4 TCP LB services, the CPU usage of the "kni_single" thread can increase during packet capture.

nsxcli.log
<TIME_STAMP> 2657104 cli.server.cli_command_service INFO {10000} CMD: set capture session 0 interface ###-###-###-###-### direction dual                                                                                                                                                                                                                                                                   
<TIME_STAMP> 2657104 cli.audit INFO CMD: set capture session 0 interface ###-###-###-###-### direction dual (duration: 0.016s), Operation status: CMD_EXECUTED                                                                                                                                                                                                                                             
<TIME_STAMP> 2657104 cli.server.cli_command_service INFO {10000} CMD: exit                                                                                                                                                                                                                                                                                                                                                  
<TIME_STAMP> 2657104 cli.audit INFO CMD: exit (duration: 0.000s), Operation status: CMD_EXECUTED_REQUESTED_EXIT                                                                                                                                                                                                                                                                                                             
<TIME_STAMP> 2657104 cli INFO NSX CLI stopped for user: admin                                                                                                                                                                                                                                                                                                                                                               
<TIME_STAMP> 2658768 cli INFO NSX CLI started (Edge) for user: admin

kern.log
<TIME_STAMP> ### kernel - - - [6955178.452519] getrandom: Device open                                                                                                                                                                                                                                                                                                                      
<TIME_STAMP> ### kernel - - - [6955178.452529] getrandom: Device close                                                                                                                                                                                                                                                                                                                     
<TIME_STAMP> ### kernel - - - [6955179.089387] rte_kni: Creating kni..

Resolution

• Please check if there is any left-over capture session by using the below command.

edge> get capture session
<TIME_STAMP>
Packet Capture Session
ID                   : 0
PORTS                : ['fp-eth0']

 

• Delete the existing capture session so that the "span-0" will not be used anymore.

edge> del capture session 0