IPFabric configured to VMware NSX receives error 'Request failed with status code 500' during discovery.

Products

• VMware NSX

Issue/Introduction

• During discovery of NSX objects using IPFabric multiple endpoint queries error within IPFabric with the status 'Request failed with status code 500'.
• Manually running the same API endpoint to the NSX manager shows no error and completes successfully.
• When checking the NSX API logs (Using KB NSX API Rate limits and logs) no errors are observed for APIs received from IPFabric.
  • Log files to check within NSX are dependant upon the NSX version of the environment,  post 4.0.1 - /var/log/proxy/envoy_access_log.txt or in a pre 4.0.1 - /var/log/proxy/localhost_access_log.txt. This file is per NSX manager node so will need to be validated on whichever node is the owner of the IP configured as the target in IPFabric.
    
    
  • This log file will look like the below format with an example line followed by what the values mean. When reviewing, 200 means the API was responded to successfully and means no issue, 503 and 419 can mean too many APIs are hitting the manager and the above KB should be reviewed in more detail.
    
    2024-10-08T10:32:31.507Z 192.168.1.100 - "GET /api/v1/transport-nodes/########-43f1-4d61-914c-############/network/interfaces/vmnic2/stats HTTP/1.1" 200 196 2891 2891
    <Timestamp> <IP of requestor (Ipfabric in this example)> - <API request type > <API Request> <HTTP response, 200 is good> <data values and time taken> 
    
    
  • So for instance to check for all log files with an error (non 200 response) you can run as root on the relevant NSX Manager node the below.  
    • grep <IP of IPfabric VM> /var/log/proxy/<envoy_access_log.txt or /var/log/proxy/localhost_access_log.txt> | grep -v 200 
    • eg "grep 192.168.0.2 /var/log/proxy/envoy_access_log.txt  | grep -v 200" 

Environment

VMware NSX
VMware NSX-T Data Center

Cause

This issue may be caused by internal API rate limiting within IPFabric.

Resolution

Engage IPFabric support if NSX API logging shows no NSX side rate limiting or errors for relevant API calls. 

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

• NSX Manager support bundles
• Text of any error messages seen in NSX GUI or command lines pertinent to the investigation
• Errors observed within IPFabric discovery GUI

Handling Log Bundles for offline review with Broadcom support

• Collect Support Bundles for Troubleshooting VMware NSX
• Uploading files to cases on the Broadcom Support Portal
• Creating and managing Broadcom support cases