NSX 4.2.x Edge memory usage continually rises when a Load Balancer is configured.
search cancel

NSX 4.2.x Edge memory usage continually rises when a Load Balancer is configured.

book

Article ID: 386421

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • VMware NSX 4.2.x is deployed.
  • Edge memory usage is observed to continually increase over time.
  • This issue can be encountered in two scenarios:

    1. A Load Balancer is configured with Cookie Garbling enabled in the persistence profile.
      • In this scenario, the "nginx: worker process" will be seen to increase over time, eg:
        Day 1
           PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+    TGID COMMAND
        12345 root      20   0   65.1g 379084  40212 R  41.2   4.9   1880:08   12345 /opt/vmware/nsx-edge/sbin/datapathd
         12346 lb        20   0  459236 251892  26244 S   0.0   3.2  42:02.69   12346 nginx: worker process
         567 root      19  -1  325332 250172 248196 S   0.0   3.2  19:39.63     567 /lib/systemd/systemd-journald

        Day 3
           PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+    TGID COMMAND
         12346 lb        20   0  620680 410612  23508 S   0.0   5.3  82:51.33   12346 nginx: worker process
        12345 root      20   0   65.1g 378404  37748 S  41.2   4.9   3694:00   12345 /opt/vmware/nsx-edge/sbin/datapathd
        6789 www-data  20   0  289296 106172  15772 S   0.0   1.4  56:26.85    6789 /usr/bin/python3

        Day 6
           PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+    TGID COMMAND
         12345 lb        20   0  899108 688152  22620 S   5.9   8.8 153:34.63   12345 nginx: worker process
        12346 root      20   0   65.1g 373320  31196 S  41.2   4.8   6792:19   12346 /opt/vmware/nsx-edge/sbin/datapathd
        6789 www-data  20   0  289296 103544  13144 S   0.0   1.3 103:03.84    6789 /usr/bin/python3


      • An Edge memory usage alarm may be triggered in the UI with log entries similar to the below observed in var/log/syslog on the NSX Manager:
        [nsx@6876 comp="nsx-edge" subcomp="node-mgmt" username="root" level="WARNING" eventFeatureName="edge_health" eventType="edge_memory_usage_high" eventSev="warning" eventState="On" entId=" "] The memory usage on Edge node <Edge_Node_ID> has reached 80% which is at or above the high threshold value of 80%.


    2. A Load Balancer virtual server is configured with Client SSL or server SSL.
      • In this scenario, the memory leak happens when the certificate is configured in client SSL or server SSL in LB virtual server.
      • It only happens when the certificate is added or updated, as the certificate update frequency is low, the impact in this scenario is minor eg:
            PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+    TGID COMMAND
         4322 lb        20   0  479272 290836 286760 S   0.0   3.8   0:00.45    4322 /opt/vmware/nsx-edge/bin/nginx
         4321 lb        20   0  506356 227416 223044 S   0.0   3.0   0:04.06    4321 nginx: LB OPER process
         4320 lb        20   0  583296  88940   7536 S   0.0   1.2   1:16.08    4320 nginx: worker process
         4561 lb        20   0  583000  85208   4108 S   0.0   1.1   1:24.42    4561 nginx: worker process
         4562 lb        20   0  583148  84768   3556 S   0.0   1.1   0:47.02    4562 nginx: worker process
         4563 lb        20   0  506356   6264   1896 S   0.0   0.1   0:00.00    4563 nginx: L4LB CP process
         4564 lb        20   0  323720   5616   1060 S   0.0   0.1   0:11.25    4564 nginx: LB CONF process

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue is encountered due to a change that was made in an updated version of Open SSL that is included in VMware NSX 4.2.

Resolution

This is a known issue impacting VMware NSX.

 

Workaround

It is possible to reduce the memory consumption by disabling Cookie Garbling in the Load Balancer cookie persistence profile.

Additional Information

For more information on Cookie Garbling, see Configure Persistent Profiles in Manager Mode