NSX 4.2.x Edge memory usage continually rises when a Load Balancer is configured.

Products

• VMware NSX

Issue/Introduction

• VMware NSX 4.2.x is deployed.
• Edge memory usage is observed to continually increase over time.
• This issue can be encountered in two scenarios:
  
  
  1. A Load Balancer is configured with Cookie Garbling enabled in the persistence profile.
     
     • In this scenario, the "nginx: worker process" will be seen to increase over time, eg:
       

       Day 1
          PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+    TGID COMMAND
        12345 root      20   0   65.1g 379084  40212 R  41.2   4.9   1880:08   12345 /opt/vmware/nsx-edge/sbin/datapathd
        12346 lb        20   0  459236 251892  26244 S   0.0   3.2  42:02.69   12346 nginx: worker process
          567 root      19  -1  325332 250172 248196 S   0.0   3.2  19:39.63     567 /lib/systemd/systemd-journald
       
       Day 3
          PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+    TGID COMMAND
        12346 lb        20   0  620680 410612  23508 S   0.0   5.3  82:51.33   12346 nginx: worker process
        12345 root      20   0   65.1g 378404  37748 S  41.2   4.9   3694:00   12345 /opt/vmware/nsx-edge/sbin/datapathd
         6789 www-data  20   0  289296 106172  15772 S   0.0   1.4  56:26.85    6789 /usr/bin/python3 
       
       Day 6
          PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+    TGID COMMAND
        12345 lb        20   0  899108 688152  22620 S   5.9   8.8 153:34.63   12345 nginx: worker process
        12346 root      20   0   65.1g 373320  31196 S  41.2   4.8   6792:19   12346 /opt/vmware/nsx-edge/sbin/datapathd
         6789 www-data  20   0  289296 103544  13144 S   0.0   1.3 103:03.84    6789 /usr/bin/python3

       
       
       
     • An Edge memory usage alarm may be triggered in the UI with log entries similar to the below observed in var/log/syslog on the NSX Manager:

       [nsx@6876 comp="nsx-edge" subcomp="node-mgmt" username="root" level="WARNING" eventFeatureName="edge_health" eventType="edge_memory_usage_high" eventSev="warning" eventState="On" entId=" "] The memory usage on Edge node <Edge_Node_ID> has reached 80% which is at or above the high threshold value of 80%.

       
       
       
  2. A Load Balancer virtual server is configured with Client SSL or server SSL.
     • In this scenario, the memory leak happens when the certificate is configured in client SSL or server SSL in LB virtual server.
     • It only happens when the certificate is added or updated, as the certificate update frequency is low, the impact in this scenario is minor eg:
       

           PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+    TGID COMMAND
          4322 lb        20   0  479272 290836 286760 S   0.0   3.8   0:00.45    4322 /opt/vmware/nsx-edge/bin/nginx 
          4321 lb        20   0  506356 227416 223044 S   0.0   3.0   0:04.06    4321 nginx: LB OPER process
          4320 lb        20   0  583296  88940   7536 S   0.0   1.2   1:16.08    4320 nginx: worker process
          4561 lb        20   0  583000  85208   4108 S   0.0   1.1   1:24.42    4561 nginx: worker process
          4562 lb        20   0  583148  84768   3556 S   0.0   1.1   0:47.02    4562 nginx: worker process
          4563 lb        20   0  506356   6264   1896 S   0.0   0.1   0:00.00    4563 nginx: L4LB CP process
          4564 lb        20   0  323720   5616   1060 S   0.0   0.1   0:11.25    4564 nginx: LB CONF process

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue is encountered due to a change that was made in an updated version of Open SSL that is included in VMware NSX 4.2.

Resolution

This issue is resolved in VMware NSX 4.2.1.3, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

It is possible to reduce the memory consumption by disabling Cookie Garbling in the Load Balancer cookie persistence profile.

Additional Information

For more information on Cookie Garbling, see Configure Persistent Profiles in Manager Mode