Virtual Servers on a small LB instance stop working when more than 10 Virtual Servers are created
Article ID: 316661
Updated On:
Products
Issue/Introduction
The following conditions are met
- NSX-T 2.4.0/2.4.1/2.4.2 environments which has been upgraded from NSX-T 2.3.x
- A small LB instance has more than 10 Virtual Servers configured
- Some Virtual Servers on the small LB are no longer functional
- If the number of Virtual Servers configured is reduced to 10 or below the problem is no longer present
- Small LB config file shows a limit of 10 L4 Virtual Servers
l4_worker_processes 1;
l4_virtual_servers 10;
l4_sessions 100000;
- Edge /var/log/syslog has log messages similar to this
<27>1 2019-12-01T19:51:11.960742+00:00 nsx-edge NSX 29259 LB [nsx@6876 comp="nsx-edge" subcomp="nsx-edge-lb.lb_log" level="ERROR" errorCode="EDG9999999"] [2187ad15-29bd-42d7-af44-6581cd47c156] [error] 29259#0: l4lb failed to find vs
Environment
Cause
Due to an issue in the upgrade process the limit of 10 is not changed to 20 in the config file.
When more than 10 Virtual Servers are configured on the LB it will result in the failure of some VIPs.
Resolution
Workaround:
If an upgrade is not possible the following workaround can be applied
Edit the content of the file /config/vmware/edge/lb/etc/lbs_small.conf as follows
l4_worker_processes 1;
l4_virtual_servers 20;
l4_sessions 105000;
For the change to take effect detach and reattach the LB from the Tier-1 Logical Router.
In a PKS environment it will not be possible to make this change from the UI due the protected principle identiy objects.
GET https://<NSX_MGR>/api/v1/loadbalancer/services/<UUID>;
To detach the LB from the Tier-1 Gateway remove the following section from the Payload and perform a PUT operation
"attachment": {
"target_id": "a3e9ec60-ad7b-45d3-82c6-cca685ffc7f3",
"target_display_name": "lb-pks-a6857d10-0c4a-42cb-a9d4-288f052c9f88-rxqnx",
"target_type": "LogicalRouter",
"is_valid": true
},
PUT https://<NSX_MGR>/api/v1/loadbalancer/services/<UUID>; using header X-Allow-Overwrite:True
To reattach the LB to the Tier-1 Gateway, perform the GET again and add back in the attachment section and again update the configuration with the PUT
PUT https://<NSX_MGR>/api/v1/loadbalancer/services/<UUID>; using header X-Allow-Overwrite:True
Example using curl:
1.) Run GET API.
curl -k -H "Content-Type:application/json" -u admin -X GET "https://<NSX-Manager-IP>/api/v1/loadbalancer/services/<Load-Balancer-uuid>"
Example Response:
{
"enabled" : true,
"relax_scale_validation" : false,
"size" : "SMALL",
"error_log_level" : "INFO",
"virtual_server_ids" : [ "b641853b-c6f2-45b3-9eef-5fe1620a108f", "4491430f-6aad-429e-92e8-6720d41ff2ff" ],
"attachment" : {
"target_id" : "a9d554c0-7faa-4014-9be8-5f8f8a389494",
"target_display_name" : "TestTier1",
"target_type" : "LogicalRouter",
"is_valid" : true
},
"resource_type" : "LbService",
"id" : "1c611398-2104-4912-8369-f1fea62857f6",
"display_name" : "Test-LB",
"tags" : [ {
"scope" : "policyPath",
"tag" : "/infra/lb-services/Test-LB"
} ],
"_create_user" : "PKS_PI",
"_create_time" : 1609365823634,
"_last_modified_user" : "PKS_PI",
"_last_modified_time" : 1611760612607,
"_system_owned" : false,
"_protection" : "REQUIRE_OVERRIDE",
"_revision" : 3
}
2.) Create a text file to alter the body of the PUT API, removing the attachment. You may also remove the _create_user , _create_time , _last_modified_user , _last_modified_time , _system_owned and , _protection field as they are not required for a PUT api call. :
vim /tmp/body
{
"enabled" : true,
"relax_scale_validation" : false,
"size" : "SMALL",
"error_log_level" : "INFO",
"virtual_server_ids" : [ "b641853b-c6f2-45b3-9eef-5fe1620a108f", "4491430f-6aad-429e-92e8-6720d41ff2ff" ],
"attachment" : {
},
"resource_type" : "LbService",
"id" : "1c611398-2104-4912-8369-f1fea62857f6",
"display_name" : "Test-LB",
"tags" : [ {
"scope" : "policyPath",
"tag" : "/infra/lb-services/Test-LB"
} ],
"_revision" : 3
}
3.) Run PUT API call with altered body to detach LB from Tier 1:
curl -k -H "Content-Type:application/json" -H "X-Allow-Overwrite:True" -u admin -d /tmp/body -X PUT "https://<NSX-Manager-IP>/api/v1/loadbalancer/services/<Load-Balancer-uuid>"
4.) Alter body text file to include Tier 1 Attachment once again, note to increase the "revision" field by a value of '1' to account for the change in step 3.
vim /tmp/body
{
"enabled" : true,
"relax_scale_validation" : false,
"size" : "SMALL",
"error_log_level" : "INFO",
"virtual_server_ids" : [ "b641853b-c6f2-45b3-9eef-5fe1620a108f", "4491430f-6aad-429e-92e8-6720d41ff2ff" ],
"attachment" : {
"target_id" : "a9d554c0-7faa-4014-9be8-5f8f8a389494",
"target_display_name" : "TestTier1",
"target_type" : "LogicalRouter",
"is_valid" : true
},
"resource_type" : "LbService",
"id" : "1c611398-2104-4912-8369-f1fea62857f6",
"display_name" : "Test-LB",
"tags" : [ {
"scope" : "policyPath",
"tag" : "/infra/lb-services/Test-LB"
} ],
"_revision" : 4
}
5.) Run the PUT API once more, with the body to re-attach the LB entity to the Tier 1 gateway it was previously attached to.
curl -k -H "Content-Type:application/json" -H "X-Allow-Overwrite:True" -u admin -d /tmp/body -X PUT "https://<NSX-Manager-IP>/api/v1/loadbalancer/services/<Load-Balancer-uuid>"
Feedback
