ESXi Firewall Rule configuration continuously updated on the hosts with enable and disable operations for "esxupdate"

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Every hour, a number of ESXi Firewall Rule configuration change events are observed on all hosts in vSAN Clusters:

Firewall configuration has changed. Operation 'enable' for rule set 'esxupdate' succeeded.
Firewall configuration has changed. Operation 'disable' for rule set 'esxupdate' succeeded.

• You also notice VUM scan events around the above Firewall changed events.

Successfully scanned <ESXi FQDN>

• Messages in envoy.log might show the local http connection limit being hit due to connections to 127.0.0.1:8084 (8084, 9084, and 9087 are VUMs port)

YYYY-MM-DDTHH:MM:SS.SSZ warning envoy[3405621] [Originator@6876 sub=filter] [C11747] local https connections exceed max allowed: 2048
YYYY-MM-DDTHH:MM:SS.SSZ warning envoy[3405621] [Originator@6876 sub=filter] [C11747] closing connection TCP<127.0.0.1:41032, 127.0.0.1:8084>

• The user initiating the VUM scan is com.vmware.vsan.health
• The  timing of these events varies with different clusters
• The vSAN clusters have vSAN Health Service enabled with a scheduled check every 60 minutes

Environment

VMware vSAN 7.0.x
VMware vSAN 6.6.x
VMware vSAN 8.0.x

Cause

vSAN generates system baselines and baseline groups for use with vSphere Update Manager. You can use these recommended baselines to update software, patches, and extensions for hosts in your vSAN cluster.

vSAN 6.6.1 and later generates automated build recommendations for vSAN clusters. vSAN combines information in the VMware Compatibility Guide and vSAN Release Catalog with information about the installed ESXi releases. These recommended updates provide the best available release to keep your hardware in a supported state.

The vSAN Build Recommendation checks are initiated as part of the vSAN Health Check that is run every 60 mins (default configuration). See KB https://knowledge.broadcom.com/external/article?legacyId=58891 for details on this test.

Resolution

This is an expected behavior based on the configuration of the vSAN Health Check in the affected vSAN cluster.
If you wish to disable the VUM scans or reduce their frequency, check the Workaround section of this article.

Workaround:
Option 1: Reduce the frequency of VUM Scans

For e.g. to set the scans to happen weekly, do the below:

• SSH into the vCenter Appliance
• Edit /etc/vmware-vsan-health/config.conf file using a text editor like 'vi'
• Under [VumIntegration] section add the below parameters (the values are in minutes, 10080=7 days)

            autoCheckerInterval = 10080
            VumScanTaskIntervalInMin = 10080

• Restart vSAN health service using this command

/usr/lib/vmware-vmon/vmon-cli -r vsan-health

Option 2: Disable vSAN Baseline VUM Scans completely

Note: Build recommendation would not be provided in vSAN Health Service after making the changes below.

• SSH into the vCenter Appliance
• Edit /etc/vmware-vsan-health/config.conf file using a text editor like 'vi'
• Under the parameter [KillSwitch], add the following field. Create one if [KillSwitch] section doesn't exist.

[KillSwitch]

             VumIntegration = false

• Restart vSAN health service using this command

/usr/lib/vmware-vmon/vmon-cli -r vsan-health