VMware Cloud Foundation Health: Ensuring Connectivity of ESX: Troubleshooting Guide

Products

VCF Operations

Issue/Introduction

The ESXi host is the fundamental block of the software-defined data center, responsible for executing workloads and interfacing directly with the underlying hardware. For a VMware Cloud Foundation (VCF) environment to remain manageable and performant, each host must maintain persistent communication with management components like vCenter Server, NSX and VCF Operations.

While administrators often rely on a host’s "Connected" status in the vCenter UI or a successful response to a ping as a sign of health, these are only surface-level indicators.

When connectivity checks between the ESXi host and VCF management components fail:

It prevents critical telemetry collection
Interrupts lifecycle management tasks
Can also lead to a loss of visibility into the physical layer

Ensuring full reachability across these specific management paths is vital to maintaining the integrity of the entire VCF software stack. Specifically, the following connectivity is required in addition to VC / ESX connectivity

NSX manager manages the NSX to install, configure, and manage NSX networking and security components on them.
VCF Operations directly collects telemetry data (metrics, health reports) directly from ESX. This direct collection eliminates load from VC and provides limited data visibility when VC is down. For direct collection, VCF Operation retrieves a limited-time token from VC. Consequently, VCF Operations connectivity to VC is also required.

To provide this reachability visibility, VCF Health checks the connectivity between following:

vCenter and ESX
VCF Operations and ESX
NSX and ESX
VCF Operations and vCenter

These checks are performed every 5 minutes.

Environment

Operations for VMware Cloud Foundation 9.1

Resolution

ESX has Failed or is Powered Down

If all connectivity checks report a failure, most often the host has failed or been explicitly powered off. Check to see whether the host is powered on and operational. For example, you can attempt to connect to the host (UI) client or ssh to it. If the host responds, proceed with the other steps listed below.

vCenter to ESX connectivity failure

When the vCenter is unable to reach ESX and the host is not down, often the problem is caused by network, certificate mismatches, or not-responsive management agents. Follow these steps to diagnose and resolve the issue:

Verify the physical host is running and connected to the network.
ESX hosts send heartbeats to vCenter on port 902 UDP. If this network flow is blocked by firewalls then the ESX host will show as Not Responding in vCenter
MTU Mismatch: Verify that the MTU settings on the ESXi Management VMkernel port match the physical switch settings (e.g., 1500 or 9000).
Check VPXA and hostd Service: the ESX diagnostic page reports on the health of these two agents. If the VCF Operations to ESX check passed, the information reported in the UI is current. If not, log into the ESXi host via SSH and restart the vCenter Agent service: /etc/init.d/vpxa restart and /etc/init.d/hostd restart.

VCF Operations to ESX connectivity failure

When the VCF Operations to ESXi check reports a failure, the problem could be due to the VC adapter not running or having failed, VCF Operations not being able to reach the collector, or the VC adapter running in the adapter not being able to reach ESX. Begin by determining whether the adapter is running. If it is, then the problem is between the VC adapter and the host.

A first step is to determine whether the same check is failing for others hosts in the VC inventory. If all hosts report this connectivity check failure, there is a systemic problem impacting all hosts. Start with step 1. In addition, if the network configuration of the host is known to be different than others, also start with step 1. If neither of these cases apply, the hardware or the management agent is likely to be causing the issue. In this case, start with step 2.

Network & DNS Validation

DNS Resolution: Resolution is done by the collector on which the adapter is running. Typically, the collector is a cloud proxy. If you are using the default dns configuration, ensure the VCF Operations nodes can resolve the ESXi Host's FQDN and that the Reverse Lookup record is accurate. If you have configured cloud proxies with different DNS servers, perform this check on the cloud proxy VM.
Port Connectivity: Validate that the following port is open from the Analytics/Collector nodes to the ESXi Management IP:

HTTPS (443): For API-based metric collection.

MTU Mismatch: Verify that the MTU settings on the ESXi Management VMkernel port match the physical switch settings (e.g., 1500 or 9000).
Ping Test: SSH into the VCF Operations node where VC adapter is running and try to ping the ESX host’s IP.

Host-side remediation

If the network is clear, the issue usually resides within the ESXi Management Agents
(hostd and vpxa).

Examine Host Resources

Check Disk Space: On the host, run df -h. If the / (root) or /tmp partitions are at 100%, the management agents cannot create the temp files needed to talk to VCF Ops.
Check Memory: If the host is in a "degraded" state due to hardware faults (purple screen of diagnostics/PSOD or lockups), it will require a hard reboot.

Restart Management Agents Restarting Management Agents in ESXi

VCF Operations to vCenter connectivity failure

Review the introduction to the previous check-failure section to understand the dependency on vCenter.

Network & DNS Validation

DNS Resolution: Repeat the resolution test listed in the previous section but for the vCenter IP address and verify that the Reverse Lookup record is accurate.
Test Bidirectional Port 443: Ensure port 443 is open between all VCF Operations nodes and the vCenter appliance.