Can not enable Supervisor with "Error message: No single Supervisor Control Plane VM provisioned successfully ["API call to VMware ESX Agent Manager (eam) failed."

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

The following unique error identifiers appear in your logs:

• The following error string is observed within the vCenter Server user interface:
  

  • A general system error occurred. Error message: No single Supervisor Control Plane VM provisioned successfully ["API call to VMware ESX Agent Manager (eam) failed. Details 'ServerFaultCode: java.lang.NullPointerException'" "API call to VMware ESX Agent Manager (eam) failed. Details 'ServerFaultCode: java.lang.NullPointerException'" "API call to VMware ESX Agent Manager (eam) failed. Details 'ServerFaultCode: java.lang.NullPointerException'"].

• The following entries are logged inside the ESX Agent Manager log file (/var/log/vmware/eam/eam.log):
  

  • |  WARN | vlsi | VcInventory.java | 369 | Tried to obtain host without parent host-#####
    |  WARN | vlsi | VcInventory.java | 437 | Could not acquire remote object for: ManagedObjectReference: type = HostSystem, value = host-xxxxxx, serverGuid = <serverGuid-Redacted>
    com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: null
            at com.vmware.eam.vc.VcInventory.addHostSystem(VcInventory.java:371) ~[eam-server.jar:?]
    :
    | ERROR | vlsi | EsxAgentManagerImpl.java | 550 | Unexpected exception:
    java.lang.NullPointerException: null
            at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011) ~[?:1.8.0_422]
            at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) ~[?:1.8.0_422]

• Concurrently, the Workload Control Plane log file (/var/log/vmware/wcp/wcpsvc.log) records a failure during agency creation:

  • error wcp [mastervm/agency.go:414] [opID=<opID-Redacted>] Failed to create new EAM agency with config types.AgencyConfigInfoEx{AgencyConfigInfo:types.AgencyConfigInfo{DynamicData:types.DynamicData{}, AgentConfig:[]types.AgentConfigInfo{types.AgentConfigInfo{DynamicData:types.DynamicData{}, ProductLineId:"", HostVersion:"", OvfPackageUrl:"https://<fqdn-ovf-package-url:5480/wcpagent/photon-ova-medium.ovf", 
    :

Environment

VMware vSphere Kubernetes Service 
vSphere with Tanzu

Cause

ESX Agent Manager (EAM) attempts to process host-#####, but the host object is missing or invalid in the inventory. This leads to a ManagedObjectNotFound condition during listener registration.
This issue can occur due to stale inventory references within the ESX Agent Manager (EAM) service memory. EAM attempts to register host-level listeners for a managed object ID (MOID) that either no longer exists in the vCenter inventory or was removed and re-added. The service crashes when it encounters this null value.

Resolution

From a vCenter SSH session, follow these steps to clear the stale inventory state and resume Supervisor provisioning:

• Restart the EAM Service - Connect to your vCenter Server via SSH and restart the EAM service to clear cached inventory references.
  Review this command before running it.

service-control --restart vmware-eam

• Verify Host Connectivity - Ensure that all ESXi hosts are visible in vCenter, connected, and has no pending HA or maintenance mode tasks.
  If you recently re-added the host, verify it is properly included in the target cluster.
• Restart the WCP Service - Once the EAM service is stable, restart the WCP service to trigger a re-synchronization of the Supervisor deployment.
  Review this command before running it.

service-control --restart wcp