Increased latency during API calls and cluster upgrade failures
search cancel

Increased latency during API calls and cluster upgrade failures

book

Article ID: 422668

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • The environment was previously upgraded from 3.1.x to 3.2.x, and subsequently to 4.x. 
  • After 3.1.x to 3.2.x upgrade, issuing API calls to retrieve logical switches/logical ports, is taking substantial amount of time to complete, or these APIs are long running.
  • This is not limited to GET API calls, it will impact other API calls such as GET, POST, DELETE related to logical switches / logical ports.
  • For example this logical switch / logical port GET API call:
    GET <NSX Manager IP>/api/v1/logical-switches
    GET <NSX Manager IP>/api/v1/logical-ports
  • Logical switchport tagging operations may also take longer time, typically more than 30 seconds.
  • Within NSX Manager's /var/log/proton/nsxapi.log, multiple association calls are seen, which in turn generate excessive RPC calls to CCP: 
    DEBUG NSX 10227 [nsx@4413 comp="nsx-manager" level="DEBUG" logger="AssociationServiceImpl" msgID="INVENTORY" reqId="########-d9c5-4486-851d-############" subcomp="manager" threadName="http-nio-127.0.0.1-7440-exec-395" username=""] Received source identifier LogicalSwitch/########-d268-############ and target type VNI true false
  • In the NSX Manager's /var/log/syslog, the following entries can be seen for the API calls with high latency: 
    [level="INFO" subcomp="manager"] UserName:<'Username'> ModuleName:'common-services' Operation:'GET@/api/v1/logical-switches' Operation status: 'failure' Error: General error has occurred.
  • The NSX proxy logs shows the delay from the initial request measured in milliseconds, in the example below, the value '1269667', 'to the upstream response.
    Log lines similar to below are seen in /var/log/proxy/envoy_access_log: 
    <source IP address> <destination IP address> "GET" "/api/v1/logical-switches" "HTTP/1.1" 200 - 0 145035 1269929 1269667 "<source IP address> "python-requests/2.31.0" "########-dbe1-4632-bcd7-############" <NSX manager FQDN> "127.0.0.1:7440"
  • When you query IpDiscoverySwitchingUpmProfile using the following REST API call, a high result_count is observed: 
    GET https://<NSX Manager ip>/api/v1/service-configs?profile_type=IpDiscoverySwitchingUpmProfile
    {
      "results" : [ ],
      "result_count" : 3852,
      "sort_by" : "display_name",
      "sort_ascending" : true
    }
  • It is observed, TKGI cluster upgrades may fail during the logical port tagging stage for the master VM.
    The failure is attributed to excessive latency, exceeding 30 seconds, in the Logical Switch Port tagging operations within the TKGI environment. 

Environment

VMware NSX

Cause

  • As part of 3.1.x to 3.2.x upgrade the service config are created for all the ports which were having IP Discovery profile attached (VIF / non-VIF).
  • Proton invokes CCP through RPC to compute the effective group membership of the LSP when a UPM profile is attached to the port.
  • Management Plane logical switch API calls (GET, PUT, DELETE), which gets the effective profile members, due to the unavailability of a service configuration. 
  • These logical port operations initiate RPC calls only when ServiceConfig objects are associated with that port.
  • ServiceConfig objects can cause unnecessary RPC calls.

Note:

  • This problem will not be seen for greenfield 4.2.x environments.
  • This issue will be seen only in environments historically upgraded from VMware NSX-T Data Center 3.1.x to VMware NSX-T Data Center 3.2.x.

Resolution

This issue is resolved in VMware NSX 4.2.4, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

Removing service configs from non-VIF will stop the excessive RPC calls to the NSX Manager.

Associate segments/logical switches with custom IP Discovery profile:

1. Log into NSX UI with admin privileges. 
2. Select Networking -> Segments -> Segment Profiles. 
3. Click Add Segment Profile and select IP Discovery. 
4. Populate and specify the IP Discovery segment profile details, matching with the 'default-ip-discovery-profile' and select 'Save'.
5. To attach the new custom IP discovery profile to the segment, go to Segment -> Click the 3 dots next to the segment where the custom profile is being applied, and select 'edit'. 
6. Expand 'Segment Profiles'. For 'IP Discovery' in the dropdown, select the newly created custom IP Discovery Profile created in step 4, and attach it to the segment. 

For more information you can review the administration guide Create an NSX IP Discovery Segment Profile.

If the workaround above can't be used, or it doesn't resolve the issue, open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.

When opening a new case, please upload the following into the support case:

  1. Support bundle from NSX Managers. Don't limit the age of the logs to be collected.
    ref. Collect Support Bundles in NSX.
  2. Provide output of the API:
    GET https://<NSX Manager ip>/api/v1/service-configs?profile_type=IpDiscoverySwitchingUpmProfile
  3. Corfu table dumps collected on NSX Manager:
    1. SSH to any NSX Manager in the cluster as root. 
    2. Run the commands below to generate database table dumps:
      /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t InternalLogicalPort > /image/InternalLogicalPort.txt
      /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t LogicalSwitch > /image/LogicalSwitch.txt
      /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t ServiceConfig > /image/ServiceConfig.txt
      /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t GroupDiscoveryProfileBindingMap > /image/GroupDiscoveryProfileBindingMap.txt
      /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t Group > /image/Group.txt
      /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t InternalGroup > /image/InternalGroup.txt
    3. Use scp, WinSCP or similar tool to extract the text files from the NSX Manager.
    4. Upload the logs, output of the API, and database table dumps to the case. 
      To upload the logs, please follow steps in KB article Uploading files to cases on the Broadcom Support Portal.

Additional Information

Powered by Wolken Software