Reconfiguration of LB in NSX-V 6.4.7 and later releases is rejected with error code 10014

Products

• VMware NSX

Issue/Introduction

When there is a change made to load balancer configuration in 6.4.7 and later versions, it may fail with the following error:

"error code 10014 : edge appliance memory is not enough to apply the configuration, pls try it later"

In the vCenter NSX UI page, you may see the error:

"errorCode":10014,"details":"Configuration failed on NSX Edge VM vm-#######. Kindly refer Edge and NSX Manager logs for more details.","rootCauseString":"edge appliance memory is not enough to apply the configuration, pls try it later.","moduleName":"vShield Edge","errorData":null

From edge logs we see

[daemon.info] INFO :: C_HAPROXY :: Executing HAPROXY feature configuration
[daemon.err] ERROR :: C_HAPROXY :: [73001] edge appliance memory is not enough to apply the configuration, pls try it later.
[daemon.err] ERROR :: VseCommandHandler :: Command failed eventually. Error: [C_HAPROXY][73001]edge appliance memory is not enough to apply the configuration, pls try it later. at /opt/vmware/vshield/Framework/lib.pm line 89

 

Environment

VMware NSX Data Center for vSphere 6.4.x

Cause

The safety check and error were added in 6.4.7 to resolve the Out Of memory and edge crash issue caused by the haproxy reconfigure.

The threshold for each edge size defined is as following:

1. Compact: 50M
2. Large threshold: 50M * 8
3. Quad threshold: 50M * 8
4. X-Large threshold: 50M * 97

If current memory is less than the threshold, the LB reconfiguration is not allowed.

Resolution

No resolution is planned.  A workaround is available.

Workaround:

1.Resize the Edge
2.Restart the Edge
3.Force kill the old haproxy process
4.Decrease the threshold defined in vshield/vse/vshield-main/Plugins/configurators/haproxy/haproxy.pm under edge root mode
In line 293-297 is the memory threshold value for each edge size defined by default.  We suggest customer change it to 10%~20% of the memory of current edge size.

293 Readonly my %HAPROXY_MAX_MEM => {"compact" => 60200,
294 "large" => 418600,
295 "quadlarge" => 418600,
296 "xlarge" => 5066400
297 };

 

Additional Information

Impact/Risks:
Customer will not be able to reconfigure the load balancer.