Users experiencing performance issues using HCX L2 extensions

Products

• VMware HCX

Issue/Introduction

• When using HCX L2 extension, users are experiencing performance issues. 
• FOU Trace shows inconsistent latency times. Sometimes shows under 2ms between peer HCX appliances. 
  • Other times this latency spikes.
  • FOUtrace command can be found by reviewing the messages log found on the NE appliance /var/log/messages.
    
• Reviewing the output from <top> command on HCX NE appliance shows 1 CPU being more utilized.
  

 

Environment

HCX 4.11.0

Cause

• In 4.11.0 there is a 1 to 1 mapping of CPU per RX queue. 
• In 4.11.0 a condition exists where an uneven amount of packets will be distributed to RX queues mapped to same CPU.
  • This lopsided distribution of packets will cause the CPU assigned to these RX queue's to become over utilized. 

Resolution

Resolution

• This issue is resolved starting in VMware HCX 4.11.1, available at Broadcom downloads.
• If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

• Enabling APR (application path resiliency) on the service mesh may provide better distribution of packets to RX queues.

Additional Information

• Seeing Bulk Migration Slowness