NSX-T Virtual Server status is unknown after upgrade to NSX-T 3.1.x

Products

• VMware NSX

Issue/Introduction

• The virtual server status is unknown.
• The virtual server does not pass traffic.
• Error seen in edge log: var/log/syslog:

2022-05-27T11:50:58.409768+00:00 edge01.corp.local NSX 8006 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="ERROR" errorCode="EDG9999999"] [########-####-####-####-##########08] init_by_lua error: error loading module 'ruleset' from file '/config/vmware/edge/lb/etc/########-####-####-####-##########08/ruleset.lua':#012#011.../lb/etc/########-####-####-####-##########08/ruleset.lua:55: invalid escape sequence near '"$appname'#012stack traceback:#012#011[C]: at 0x6d6d13c94380#012#011[C]: in function 'require'#012#011init_by_lua:3: in main chunk

• In the edge node as admin checking the LB status shows the following:

> get load-balancer ########-####-####-####-##########08 status
Fri May 27 2022 UTC 11:12:49.104
24304: Internal Error: Query LB Engine Failed.

• In the edge node as admin checking the LB snat pools shows the following:

> get load-balancer ########-####-####-####-##########08 snat-pools
Fri May 27 2022 UTC 11:14:39.070
% An unexpected error occurred: LB service is not running currently

• In the edge node as admin checking the LB diagnosis shows the following:

> get load-balancer ########-####-####-####-##########08 diagnosis
Fri May 27 2022 UTC 11:53:23.950
Checking
Action          : checking system
Result          : passed

Action          : checking crash
Result          : passed

Action          : checking daemon status
Result          : passed

Action          : checking configuration
Result          : not_found_in_db: LbCurrentMsg -> this error is on the impacted LB.

• Not all load balancers are impacted.
• Reboot of edge node does not resolve the issue.
• Replacing virtual server does not resolve the issue.

Environment

VMware NSX-T Data Center

 

You may have multiple load balancers with virtual server(s) attached.

 

You recently upgraded from 2.x to 3.1.x version of NSX-T.

Cause

It is possible to create a load balancer "HTTP_REQUEST_REWRITE" rule using REGEX conditions.
In these conditions there may be an unhandled special character in the header value.
Such an example would be:

"header_value": "$newname\\.$_server_addr" 

In the above example the '\' causes the load balancer to become unknown.
This issue was introduced when a newer version of nginx was embedded in VMware NSX-T Data Center from 3.1.x onwards.

Resolution

Remove the special character from REGEX condition in the Load balancer rule.