NSX-T Virtual Server status is unknown after upgrade to NSX-T 3.1.x
book
Article ID: 322545
calendar_today
Updated On:
Products
VMware NSX
Issue/Introduction
Symptoms:
You may have multiple load balancers with virtual server(s) attached.
The virtual server status is unknown.
The virtual server does not pass traffic.
You recently upgraded from 2.x to 3.1.x version of NSX-T.
Error seen in edge log: var/log/syslog:
2022-05-27T11:50:58.409768+00:00 edge01.corp.local NSX 8006 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="ERROR" errorCode="EDG9999999"] [########-####-####-####-##########08] init_by_lua error: error loading module 'ruleset' from file '/config/vmware/edge/lb/etc/########-####-####-####-##########08/ruleset.lua':#012#011.../lb/etc/########-####-####-####-##########08/ruleset.lua:55: invalid escape sequence near '"$appname'#012stack traceback:#012#011[C]: at 0x6d6d13c94380#012#011[C]: in function 'require'#012#011init_by_lua:3: in main chunk
In the edge node as admin checking the LB status shows the following:
cnnsxtedge01> get load-balancer ########-####-####-####-##########08 status Fri May 27 2022 UTC 11:12:49.104 24304: Internal Error: Query LB Engine Failed.
In the edge node as admin checking the LB snat pools shows the following:
cnnsxtedge01> get load-balancer ########-####-####-####-##########08 snat-pools Fri May 27 2022 UTC 11:14:39.070 % An unexpected error occurred: LB service is not running currently
In the edge node as admin checking the LB diagnosis shows the following:
cnnsxtedge01> get load-balancer ########-####-####-####-##########08 diagnosis Fri May 27 2022 UTC 11:53:23.950 Checking Action : checking system Result : passed
Action : checking crash Result : passed
Action : checking daemon status Result : passed
Action : checking configuration Result : not_found_in_db: LbCurrentMsg -> this error is on the impacted LB.
Not all load balancer's are impacted.
Reboot of edge node does not resolve the issue.
Replacing virtual server does not resolve the issue.
Environment
VMware NSX-T Data Center
Cause
It is possible to create a load balancer "HTTP_REQUEST_REWRITE" rule using REGEX conditions. In these conditions there may be an unhandled special character in the header value. Such an example would be:
"header_value": "$newname\\.$_server_addr"
In the above example the '\' causes the load balancer to become unknown. This issue was introduced when a newer version of nginx was embedded in NSX-T from 3.1.x onwards.
Resolution
Remove the special character from REGEX condition in the Load balancer rule.