Network Connectivity Loss due to physical firewall in a hung state or restarting

Products

• VMware vSphere ESXi

Issue/Introduction

You observe the following symptoms in your environment:

• On a scheduled basis, the infrastructure experiences a surge in utilization.
• The physical firewall becomes unresponsive and automatically restarts.
• Users experience a total loss of network connectivity and session slowness during this window.
• Traffic graphs confirm a massive spike, primarily concentrated in specific applications / traffic types (example: https, SMB) .
• ESXi host networking does not show saturation.

Environment

VMware vSphere ESXi

Cause

The underlying cause is a scheduled "storm" of network traffic originating from within the Horizon virtual desktops. This is typically triggered by synchronized automated tasks, such as weekly security scans, antivirus signature updates, backups, or system-wide patch checks, that occur simultaneously across the environment. This synchronization saturates the firewall's session capacity, leading to session-table exhaustion or CPU saturation on the network appliance. This is not a VMware issue but the effect may occur due to the volume of network traffic and the physical firewalls capacity to process applied volume. 

Resolution

To resolve this issue, you must reduce the concurrent pressure on the network appliance by de-synchronizing automated guest-level tasks.

1. Audit Scheduled Tasks: Identify any security or maintenance jobs configured to run across the Horizon guest images during the timeframe.
2. Implement Randomized Start Times: Configure a "randomized start time" for these tasks or stagger them over a wider time window. This flattens the peak "burst" of sessions, allowing the firewall to process traffic within its design limits.
3. Verify Firewall Sizing: Consult with your firewall vendor to verify if the current appliance throughput and session limits are appropriate for the peak load of the environment.
4. Identify Top Talkers: Work with your network team to identify specific source/destination IPs creating the most traffic during the surge to determine if specific workloads can be moved to a different time.

Additional Information

For additional guidance on troubleshooting network performance, see

Troubleshooting network performance issues in a vSphere environment