NSX-T Load balancer nginx service crashes frequently with core dumps due to presence of LB "rule-log" configuration.

Products

• VMware NSX

Issue/Introduction

Load balancer status is failing with LB engine query

edge01.your.domain> get load-balancer b7c148e4-###-##########2327d status
Tue Oct 29 2024 UTC 08:02:47.484
24304: Internal Error: Query LB Engine Failed.

Lb-service is not running.

edge01.your.domain> get load-balancer b7c148e4-###-##########2327d health-check-table
Tue Oct 29 2024 UTC 08:02:58.924
% An unexpected error occurred: LB service is not running currently

Load balancer service is unavailable and generates frequent nginx core dumps.

edge01.your.domain> get load-balancer b7c148e4-###-##########2327d diagnosis

Checking
Action : checking system
Result : passed

Action : checking crash
Result : core file found
               core.nginx.1730188987.278549.134.11.gz is generated.
                core.nginx.1730188987.278531.134.11.gz is generated.

Action : checking daemon status
Result : passed

Action : checking configuration
Result : not_found_in_db: LbCurrentMsg

Segmentation fault errors are found in Edge's /var/log/kern.log:
kernel.log:

2024-10-29T19:22:00.557Z edge01.your.domain.com kernel - - - [48535.297668] nginx[1667454]: segfault at 0 ip 000074c26e0e4bc2 sp 00007880f4ec9910 error 4 in libprotobuf-c.so.1.0.0[74c26e0e4000+6000]
2024-10-29T19:22:00.558Z edge01.your.domain.com kernel - - - [48535.962629] nginx[1667475]: segfault at 0 ip 000074c26e0e4bc2 sp 00007880f4ec9910 error 4 in libprotobuf-c.so.1.0.0[74c26e0e4000+6000]

nsxcli.log shows the rule-log is enabled

cli.descriptors.cli_command_service INFO CMD: set load-balancer b7c148e4-###-##########2327d rule-log virtual-server 988c4c91-#########-cde43fadd196 enabled
cli.audit INFO CMD: set load-balancer b7c148e4-###-##########2327d  rule-log virtual-server 988c4c91-#########-cde43fadd196 enable, Operation status: CMD_EXECUTED

Environment

VMware NSX

VMware NSX-T Data Center

Cause

If a rule-log is enabled for the LB virtual server and it is not disabled, before the virtual server or the pool in this LB service is detach/deleted.
And once the docker service for lb restarts( or edge reboots) we may see the nginx crashes happening frequently due to the LB rule log debug command is in place.

 

  

Resolution

Note: Edge CLI command "set load-balancer <uuid> rule-log ..." is only used for internal debug.  This rule-log has nothing to do with debug level logging with load-balancer and virtual server logging.

Please make sure that rule-log debug is disabled before the virtual server or the pool in this LB service is detach/deleted.

If you encounter the above issue with please contact to Broadcom support.