Virtual Servers on a small LB instance stop working when more than 10 Virtual Servers are created
search cancel

Virtual Servers on a small LB instance stop working when more than 10 Virtual Servers are created

book

Article ID: 316661

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:
The following conditions are met
 
  • NSX-T 2.4.0/2.4.1/2.4.2 environments which has been upgraded from NSX-T 2.3.x
  • A small LB instance has more than 10 Virtual Servers configured
  • Some Virtual Servers on the small LB are no longer functional
  • If the number of Virtual Servers configured is reduced to 10 or below the problem is no longer present
  • Small LB config file shows a limit of 10 L4 Virtual Servers
          cat /config/vmware/edge/lb/etc/lbs_small.conf
            l4_worker_processes 1;
            l4_virtual_servers  10;
            l4_sessions         100000;
  • Edge /var/log/syslog has log messages similar to this
        <27>1 2019-12-01T19:50:57.276311+00:00 nsx-edge NSX 2310 LB [nsx@6876 comp="nsx-edge" subcomp="nsx-edge-lb.lb_log" level="ERROR" errorCode="EDG9999999"] [2187ad15-29bd-42d7-af44-6581cd47c156] [error] 2310#0: l4lb failed to allocate vs - max services
       
        <27>1 2019-12-01T19:51:11.960742+00:00 nsx-edge NSX 29259 LB [nsx@6876 comp="nsx-edge" subcomp="nsx-edge-lb.lb_log" level="ERROR" errorCode="EDG9999999"] [2187ad15-29bd-42d7-af44-6581cd47c156] [error] 29259#0: l4lb failed to find vs

Environment

VMware NSX-T Data Center 2.x

Cause

NSX-T 2.3 supports 10 Virtual Servers per Small Load Balancer. NSX-T 2.4 supports 20 Virtual Servers per Small Load Balancer.
Due to an issue in the upgrade process the limit of 10 is not changed to 20 in the config file.
When more than 10 Virtual Servers are configured on the LB it will result in the failure of some VIPs.

Resolution

This issue is resolved in NSX-T Data Center 2.4.3 and 2.5.0

Workaround:

If an upgrade is not possible the following workaround can be applied
       
Edit the content of the file /config/vmware/edge/lb/etc/lbs_small.conf as follows


        l4_worker_processes 1;
        l4_virtual_servers  20;
        l4_sessions         105000;

For the change to take effect detach and reattach the LB from the Tier-1 Logical Router.

In a PKS environment it will not be possible to make this change from the UI due the protected principle identiy objects.

GET https://<NSX_MGR>/api/v1/loadbalancer/services/<UUID>;

To detach the LB from the Tier-1 Gateway remove the following section from the Payload and perform a PUT operation

  "attachment": {
    "target_id": "a3e9ec60-ad7b-45d3-82c6-cca685ffc7f3",
    "target_display_name": "lb-pks-a6857d10-0c4a-42cb-a9d4-288f052c9f88-rxqnx",
    "target_type": "LogicalRouter",
    "is_valid": true
  },

PUT https://<NSX_MGR>/api/v1/loadbalancer/services/<UUID>; using header X-Allow-Overwrite:True

To reattach the LB to the Tier-1 Gateway, perform the GET again and add back in the attachment section and again update the configuration with the PUT

PUT https://<NSX_MGR>/api/v1/loadbalancer/services/<UUID>; using header X-Allow-Overwrite:True

 

Example using curl:

1.) Run GET API.


curl -k -H "Content-Type:application/json" -u admin -X GET "https://<NSX-Manager-IP>/api/v1/loadbalancer/services/<Load-Balancer-uuid>"

Example Response:

{
  "enabled" : true,
  "relax_scale_validation" : false,
  "size" : "SMALL",
  "error_log_level" : "INFO",
  "virtual_server_ids" : [ "b641853b-c6f2-45b3-9eef-5fe1620a108f", "4491430f-6aad-429e-92e8-6720d41ff2ff" ],
  "attachment" : {
    "target_id" : "a9d554c0-7faa-4014-9be8-5f8f8a389494",
    "target_display_name" : "TestTier1",
    "target_type" : "LogicalRouter",
    "is_valid" : true
  },
  "resource_type" : "LbService",
  "id" : "1c611398-2104-4912-8369-f1fea62857f6",
  "display_name" : "Test-LB",
  "tags" : [ {
    "scope" : "policyPath",
    "tag" : "/infra/lb-services/Test-LB"
  } ],
  "_create_user" : "PKS_PI",
  "_create_time" : 1609365823634,
  "_last_modified_user" : "PKS_PI",
  "_last_modified_time" : 1611760612607,
  "_system_owned" : false,
  "_protection" : "REQUIRE_OVERRIDE",
  "_revision" : 3
}

 

2.) Create a text file to alter the body of the PUT API, removing the attachment.  You may also remove the _create_user , _create_time , _last_modified_user , _last_modified_time ,  _system_owned and , _protection field as they are not required for a PUT api call.  :

 

vim /tmp/body

{
  "enabled" : true,
  "relax_scale_validation" : false,
  "size" : "SMALL",
  "error_log_level" : "INFO",
  "virtual_server_ids" : [ "b641853b-c6f2-45b3-9eef-5fe1620a108f", "4491430f-6aad-429e-92e8-6720d41ff2ff" ],
  "attachment" : {
  },
  "resource_type" : "LbService",
  "id" : "1c611398-2104-4912-8369-f1fea62857f6",
  "display_name" : "Test-LB",
  "tags" : [ {
    "scope" : "policyPath",
    "tag" : "/infra/lb-services/Test-LB"
  } ],
  "_revision" : 3
}

 

3.) Run PUT API call with altered body to detach LB from Tier 1:

 

curl -k -H "Content-Type:application/json" -H "X-Allow-Overwrite:True" -u admin  -d /tmp/body -X PUT "https://<NSX-Manager-IP>/api/v1/loadbalancer/services/<Load-Balancer-uuid>"


4.) Alter body text file to include Tier 1 Attachment once again, note to increase the "revision" field by a value of '1' to account for the change in step 3.
 

vim /tmp/body

{
  "enabled" : true,
  "relax_scale_validation" : false,
  "size" : "SMALL",
  "error_log_level" : "INFO",
  "virtual_server_ids" : [ "b641853b-c6f2-45b3-9eef-5fe1620a108f", "4491430f-6aad-429e-92e8-6720d41ff2ff" ],
  "attachment" : {
    "target_id" : "a9d554c0-7faa-4014-9be8-5f8f8a389494",
    "target_display_name" : "TestTier1",
    "target_type" : "LogicalRouter",
    "is_valid" : true
  },
  "resource_type" : "LbService",
  "id" : "1c611398-2104-4912-8369-f1fea62857f6",
  "display_name" : "Test-LB",
  "tags" : [ {
    "scope" : "policyPath",
    "tag" : "/infra/lb-services/Test-LB"
  } ],
  "_revision" : 4
}


5.) Run the PUT API once more, with the body to re-attach the LB entity to the Tier 1 gateway it was previously attached to.

curl -k -H "Content-Type:application/json" -H "X-Allow-Overwrite:True" -u admin  -d /tmp/body -X PUT "https://<NSX-Manager-IP>/api/v1/loadbalancer/services/<Load-Balancer-uuid>"