NSX-T Virtual Server status is unknown after upgrade to NSX-T 3.1.x

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You may have multiple load balancers with virtual server(s) attached.
• The virtual server status is unknown.
• The virtual server does not pass traffic.
• You recently upgraded from 2.x to 3.1.x version of NSX-T.
• Error seen in edge log: var/log/syslog:

2022-05-27T11:50:58.409768+00:00 edge01.corp.local NSX 8006 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="ERROR" errorCode="EDG9999999"] [77886677-4004-4b99-9483-a4022b8f6408] init_by_lua error: error loading module 'ruleset' from file '/config/vmware/edge/lb/etc/77886677-4004-4b99-9483-a4022b8f6408/ruleset.lua':#012#011.../lb/etc/77886677-4004-4b99-9483-a4022b8f6408/ruleset.lua:55: invalid escape sequence near '"$appname'#012stack traceback:#012#011[C]: at 0x6d6d13c94380#012#011[C]: in function 'require'#012#011init_by_lua:3: in main chunk

• In the edge node as admin checking the LB status shows the following:

cnnsxtedge01> get load-balancer 77886677-4004-4b99-9483-a4022b8f6408 status
Fri May 27 2022 UTC 11:12:49.104
24304: Internal Error: Query LB Engine Failed.

• In the edge node as admin checking the LB snat pools shows the following:

cnnsxtedge01> get load-balancer 77886677-4004-4b99-9483-a4022b8f6408 snat-pools
Fri May 27 2022 UTC 11:14:39.070
% An unexpected error occurred: LB service is not running currently

• In the edge node as admin checking the LB diagnosis shows the following:

cnnsxtedge01> get load-balancer 77886677-4004-4b99-9483-a4022b8f6408 diagnosis
Fri May 27 2022 UTC 11:53:23.950
Checking
Action          : checking system
Result          : passed

Action          : checking crash
Result          : passed

Action          : checking daemon status
Result          : passed

Action          : checking configuration
Result          : not_found_in_db: LbCurrentMsg -> this error is on the impacted LB.

• Not all load balancer's are impacted.
• Reboot of edge node does not resolve the issue.
• Replacing virtual server does not resolve the issue.

Environment

VMware NSX-T Data Center

Cause

It is possible to create a load balancer "HTTP_REQUEST_REWRITE" rule using REGEX conditions.
In these conditions there may be an unhandled special character in the header value.
Such an example would be:

"header_value": "$newname\\.$_server_addr" 

In the above example the '\' causes the load balancer to become unknown.
This issue was introduced when a newer version of nginx was embedded in NSX-T from 3.1.x onwards.

Resolution

Remove the special character from REGEX condition in the Load balancer rule.

Workaround:
Remove the special character from REGEX condition in the Load balancer rule.