Alert "VM Network has dropped transmit packets" is triggering after applying Hot Fix 5 patch

Products

• VMware Aria Suite

Issue/Introduction

You applied HF5 and received multiple critical or immediate alerts "VM network has dropped transmit packets" on Virtual Machines.

Environment

VMware Aria Operations 8.18.3 Hot Fix 5

Cause

• The collection of the "Network|Total Transmitted Packets Dropped" metric from vCenter was not working correctly in previous versions of Aria Operations.
• Hot Fix 5 fixed the collection of the "Network|Total Transmitted Packets Dropped" metric from vCenter
  • See VMware Aria Operations 8.18 Hot Fix 5
• The "VM Network has dropped transmit packets" alert is triggered when the "Network|Total Transmitted Packets Dropped" metric exceeds the defined thresholds.
• Note: Packets that are filtered by NSX Distributed Firewall (DFW) rules are included in the "Network|Total Transmitted Packets Dropped" metric This can cause the alert to trigger for VMs with DFW rules applied.
  • See TX packet drops reported by Aria Operations for virtual machine on NSX-T segment

Resolution

Verify the Metrics Collected by Aria Operations from vCenter

1. Log in to the vSphere console and browse to the VM
2. Click on the Monitor > Advanced tab
3. Click Chart Options
4. In the Chart Metrics: click Network and place a check in the box for the Counter named Transmit packets dropped
5. Review the chart at the same time frame that the alert was triggered in Aria Operations
   
   Note: vCenter creates data points every 20 seconds for the Real-time graph. Aria Operations collects data from the vCenter every 5 minutes (by default). This results in a data point in Operations that represents the sum of 15 of the 20 second vCenter data points.

 

Workaround 1 - Increase the Alert Wait Cycles

Increasing the Wait Cycle for the alert will cause the alert to be less sensitive to transient events. The alert will not trigger unless the dropped transmit packets are occurring above the threshold for a consecutive number of collection cycles.

1. Log in to the Aria Operations UI and browse to Operations > Configurations > Alert Definitions
2. Type VM network has dropped transmit packets in the filter in the top right and press Enter
3. Click the vertical 3 dot button and select Edit
4. In the 1 - Alert tab, expand Advanced Settings
5. Change the Wait Cycle to 3 and click Update

Alert Definition Best Practices - Wait and Cancel Cycle

Creating Alert Definitions - Wait Cycle

 

Workaround 2 - Increase the Alert Threshold

The alert threshold values of 10 and 20 dropped packets are suggested values that may not be appropriate for your environment. Use the steps below to update the alert thresholds with your desired values.

1. Log in to the Aria Operations UI and browse to the VM that raised the alert
2. Click the Policy dropdown menu and click View Policy
3. Click Edit Policy
4. Click the Alerts and Symptoms tile
5. Type VM network has dropped transmit packets in the filter in the top right and press Enter
6. Select the VM network has dropped transmit packets alert definition to reveal the settings
7. Modify the values for the Immediate and Critical level as appropriate for your environment
8. Click Save
9. Repeat steps 1-8 for any remaining policies in the environment that should be modified

Note: The recommended values >20 for Immediate level and >10 for Critical level are reversed. This will be fixed in an upcoming version of the product. The steps in Workaround 2 can be used to correct the values to the intended values of >10 for Immediate level and >20 for Critical.

Additional Information