NSX-T Native Load Balancer and Virtual Server status is Unknown due to Invalid character in the LB application rule

Products

• VMware NSX

Issue/Introduction

• The NSX-T Native Load Balancer Status is unknown
• The Virtual Server status is unknown.
• All Virtual Servers on the affected Load Balancer does not pass traffic.
• Below message is seen when we try to get the LB status from Edge CLI
  nsxedge-testlb> get load-balancer <LB-UUID> status
24304: Internal Error: Query LB Engine Failed.
• Below Message is seen when running the "get load-balancer <LB-UUID> diagnosis" Command from the Edge CLI
  nsxedge-testlb> get load-balancer <LB-UUID> diagnosis
Checking
Action : checking system
Result : passed
Action : checking crash
Result : passed
Action : checking daemon status
Result : passed
Action : checking configuration
Result : not_found_in_db: LbCurrentMsg <<<<<<<<<<<
• Below Error Messages are seen in the Syslog Messages In the Edge (/var/log/syslog)
  2024-05-26T14:04:01.042Z  <nsxedge-testlb>NSX 60279 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="ERROR" errorCode="EDG9999999"] [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxb648] init_by_lua error: error loading module 'ruleset' from file '/config/vmware/edge/lb/etc/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxb648/ruleset.lua':#012#011.../lb/etc/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxb648/ruleset.lua:62: invalid escape sequence near '"rspirep ^Location:'#012stack traceback:#012#011[C]: at 0x735bd2bf0630#012#011[C]: in function 'require'#012#011init_by_lua:3: in main chunk

Environment

VMWare NSX

 

 

Cause

When a LB application rule gets configured with invalid character in the Action field, this new configuration does not takes effect and the LB would still use the old configuration.  At this point there is no issues with LB packet processing.
But when one of the following conditions occur:

• the Edge Node gets rebooted
• the Edge node enters and leaves the maintenance mode
• the LB instance is detached from the T1 and re-attached

the old configuration would be cleared, and the new configuration with the invalid LB rule tries to load but fails.

At this state the LB cannot process the packets anymore and the status of LB, Virtual Server would be Unknown in the NSX-T UI.

 

For example, if the action field of a LB application rule has a Invalid escape Sequence like "\" then any new LB configuration would not take affect till the LB rules are reconfigured with the valid escape sequences "\\" in this case.

Resolution

While configuring LB application rule, make sure to use valid escape sequence.

Refer to the lua manual for the valid escape sequences.

https://www.lua.org/manual/5.1/manual.html#2.1

"Literal strings can be delimited by matching single or double quotes, and can contain the following C-like escape sequences:
'\a' (bell), '\b' (backspace), '\f' (form feed), '\n' (newline), '\r' (carriage return), '\t' (horizontal tab), '\v' (vertical tab), '\\' (backslash), '\"' (quotation mark [double quote]), and '\'' (apostrophe [single quote]).
Moreover, a backslash followed by a real newline results in a newline in the string. A character in a string can also be specified by its numerical value using the escape sequence \ddd, where ddd is a sequence of up to three decimal digits.
Note: If a numerical escape is to be followed by a digit, it must be expressed using exactly three digits.

 

Strings in Lua can contain any 8-bit value, including embedded zeros, which can be specified as '\0'."