NSX Log Capture & Support Engagement Best Practices

Products

VMware NSX

Issue/Introduction

Overview
VMware Technical Support routinely requests diagnostic information from you when a support request is handled. This diagnostic information contains product specific logs, configuration files, and data appropriate to the situation for analysis.

Environment

VMware NSX-T
VMware NSX Data Center for vSphere 6.4.x
VMware NSX-T Data Center

Resolution

Important logs for VMware NSX for vSphere: Below table provide examples for information on logs that are important to debug various NSXv Services / Issues
NOTE: NSXv have already reached “End of General Support” on Jan 16^th 2022. Please refer vmware Product lifecycle Matrix

NSX Components Services	Manager	Controller	Hyper- visor	Edge
Logical Switching
Logical Routing
Distributed Firewall
Edge Services

For log collection please refer KB article: https://kb.vmware.com/s/article/2074678
VMware recommends using syslog for long term retention of logs. All NSX components support syslog. Since the disk space on each appliance (NSX Manager, NSX Controller and NSX Edge) is limited by the VM size, the log rotation policy is primarily based on size (and not based on time).

Important logs for VMware NSX-T Data Center
Below table provide examples for information on logs that are important to debug various NSX-T Services / Issues

NSX Components Services	NSX Manager	Hypervisor	Edge
Install/Upgrade
Logical Switching
Routing
Firewall
Edge Services

Doc: Please refer NSX-T Admin Guide for logging level details

Other important logs & data
Apart from the above basic logs mentioned above, VMware’s Global Support can ask for specific set of logs based on the type of issue and situation.

Debug logs (in specific scenarios, sometimes the logging level may need to be changed from ‘info’ to ‘debug’ for various components to collect more granular levels of logs. NOTE: This logging level is not enabled by default and should be enabled only when required.
Logs related to other services like Guest Introspection / Service Insertion (SVM) etc. may also be required for investigating certain issues
Packet Captures: Time to time GS will need packet captures to understand the flow. However, customers can also perform with the help of GS or by referring to multiple blogs available. For detailed steps please refer the below documents.
ADF Data: It is a script to capture performance data (netstat and vmkstat mainly). GS can provide this to the customer based on requirement to capture relevant data specially related to performance.
- NOTE: There is no public facing link for this script as it is mainly for internal use by VMware to diagnose the issue.

Additional Information

Information to be kept handy when engaging GS for Severity 1 issue

Clear & precise problem statement (eg: Production server or other mission critical system(s) are down, and no workaround is immediately available)
Business impact (eg: Your business operations have been severely disrupted)
Need to have access to all affected and related systems for troubleshooting
Exact date and time of the issue
Details of recent changes made in the environment
Data path details, topology etc.

RCA Expectation & Log Rollover
Troubleshooting operations such as ‘Reboot’ or ‘Redeploy’ can lead to logs being rolled over. This is applicable for components such as Edge/DLR, NSX Mangers, Controllers etc.
It is recommended that logs are collected before performing any intrusive operations. GS will need these for investigations when you are looking for potential triggers to the issue.
However, collecting logs requires time hence customer will need to take the decision if they are ok to wait or want GSS to immediately take steps in restoring the production.
NOTE: Incase the logs are not collected and operations like Reset / Reboot etc. are performed, it will restrict GS to provide any conclusive RCA.
Configuring Syslog
VMware recommends using syslog for long term retention of the logs as all NSX components support it. As the disk on the appliances, such as NSX Manager, NSX Controller, NSX Edge are limited by the virtual machine sizing, the log rotation policy is primarily based on size (and not based on time).
For more information, see the Configure Remote Syslog Servers section in the NSX Administration Guide.
NSX-T: Configuring Remote Logging
NSXv: Configuring a Syslog Server

Monitoring tools through NSX
NSX provides comprehensive monitoring tools through NSX native monitoring capability and integration with 3rd party tools. This includes inbuilt dashboard, Alarms/Events, Troubleshooting Alerting etc. Below are few VMware solutions which can give more insight into the environment including customized alert and recommendations

vRealize Network Insight: Allows you to gain visibility into your applications and construct meaningful insights with intelligent application discovery, network optimization, analytics, and troubleshooting.
vRealize Log Insight: With Log Insight NSX Content Pack it can provide intelligent log management for infrastructure and applications
For more details refer: NSX-T Operation Guide

Automation of logs upload
To simplify and expedite the log upload process, we recommend using the Log Assist feature of VMware Skyline, which (with your permission) automatically uploads log files. Skyline is available to all Production Support and Premier Services customers.

Attachments

adf get_app