NSX Load Balancer Service and all Virtual Servers go Down after adding a new Virtual Server

Products

• VMware NSX

Issue/Introduction

 - After adding a new Virtual Server to an existing Load Balancer Service (LBS), the LBS status changed to Down, and all underlying Virtual Servers also became unavailable.

 - Existing Virtual Servers under the same LB service were also disconnected/unreachable.

 - The impacted LB Service was created before NSX 3.2 through Policy API,

 - On the Manager UI, the admin status of the LB became Down. 

 

Environment

VMware NSX 4.x

Cause

This is caused by a realization behavior change for LB services created/updated via Policy API using {"enabled": null}, before vs after NSX 3.2.

 - NSX 3.1 behavior (Policy → MP realization path used MP API):

• The problematic LB service was originally created in 3.1 via Policy API with {"enabled": null}.
• When Policy LB had "enabled": null, the MP LB was realized as "enabled": true via MP API.
• As a result, Policy UI could show Admin Status: No, but the LB was actually enabled in MP and on the Edge, creating an inconsistency in 3.1.

 

 - NSX 3.2 and later behavior (no MP API realization for LB):

• From 3.2, LB no longer uses MP API for MP realization.
• If LB service is created/updated by Policy API with {"enabled": null}, the system directly sets the MP LB object as "enabled": null in corfu.
• On the Edge, "enabled": null is treated as enabled: false, so Policy/MP/Edge become consistently disabled.

 

Why it surfaced after upgrade and configuration change:

If you upgrade NSX from 3.1 → 3.2 or above(4.x), after upgrade, the LB will keep working because the historical state remained such that Policy LB = enabled:null while MP LB = enabled:true (still effectively enabled).

When you add a new Virtual Server, it triggers LB service realization.

In 4.1, the Policy LB configuration of enabled:null is realized as enabled:null in MP, which is effectively disabled on the Edge, causing the MP LB service to transition from enabled → disabled, and the LB service (and underlying Virtual Servers) to go down.

Resolution

If you encountered this issue, please:

1) perform a full inventory check of the LB service status across the entire NSX environment using the command below:

 - curl -k -u 'admin' -H 'Content-Type:application/json' -X GET https://<mgr-ipaddress>/policy/api/v1/infra/lb-services 

2) For each LB Service, verify:

    a) whether the "enabled" field exists, and

    b) if it exists, whether the value is set to "true".

If the value is "true", that LB is not expected to encounter the same issue during the operation(modifying LB configuration).

 

3) If the "enabled" field is missing, or if the value is not "true" (e.g., it is "null"), the same problem may occur.

 

Perform the following steps for any LB Service where "enabled": true is not confirmed:

 - curl -k -u 'admin' -H 'Content-Type:application/json' -X GET https://<NSX IP>/policy/api/v1/infra/lb-services/<LBID>

 - Create a test.json file using the output of the command above, add the enabled value, and then run the command below:

     - Line to add: "enabled" : true,

     - curl -k -i -u 'admin' -H 'Content-Type:application/json' -X PATCH https://<NSX IP>/policy/api/v1/infra/lb-services/<LBID> -d @./test.json

     - test.json file example:

{   "connectivity_path" : "/infra/tier-1s/test",   "enabled" : true,   "relax_scale_validation" : false,   "size" : "SMALL",   "error_log_level" : "INFO",   "resource_type" : "LBService",   "id" : "test",   "display_name" : "test",   "path" : "/infra/lb-services/test",   "relative_path" : "test",   "parent_path" : "/infra",   "remote_path" : "",   "unique_id" : "####",   "realization_id" : "####",   "owner_id" : "####",   "marked_for_delete" : false,   "overridden" : false,   "_system_owned" : false,   "_protection" : "NOT_PROTECTED",   "_create_time" : ####,   "_create_user" : "admin",   "_last_modified_time" : ####,   "_last_modified_user" : "admin",   "_revision" : 10 }