Error "A general system error occurred host management agents not reachable" when adding host to vCenter
search cancel

Error "A general system error occurred host management agents not reachable" when adding host to vCenter

book

Article ID: 408920

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • When adding an ESXi host to vCenter or reconnecting it post-upgrade, the process may fail with the error: 

 "A general system error occurred host management agents not reachable"

or 

"A general system error occurred: Timed out waiting for vpxa to start." 

  • vCenter logs (/var/log/vmware/vpxd/vpxd,log) indicate existing users and policy applied/exists 

YYYY-MM-DDThh:mm:ss.msZ info vpxd[06373] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=5f2770bf] [PolicyUpdater] Grants for client profile LU###vpxuser on host <Host FQDN> are already in desired state
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[06373] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=5f2770bf] [CreateOrUpdate] Policy with external group name TrustedAdmins already exists on host <Host FQDN>. Exception: N4Vpxd7Clients22AlreadyExistsExceptionE(Error:
-->    com.vmware.vapi.std.errors.already_exists

YYYY-MM-DDThh:mm:ss.msZ info vpxd[06373] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=5f2770bf] [PolicyUpdater] Grants for client profile SG#https://<vCenter FQDN>/websso/SAML2/Metadata/vsphere.local#vsphere.local#TrustedAdmins on host <Host FQDN> are already in desired state
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[06373] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=5f2770bf] [CreateOrUpdate] Policy with external group name vStatsGroup already exists on host <Host FQDN>. Exception: N4Vpxd7Clients22AlreadyExistsExceptionE(Error:
-->    com.vmware.vapi.std.errors.already_exists
--> No messages!

  • vpxa agent logs (/var/run/log/vpxa.log) on ESXi indicates backtrace when nvme device being queried and resulting in the service crashing down. 

YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101241]: [Originator@6876 sub=vmomi.soapStub[5] opID=mats8od4-250817-auto-5dj8-h5:70029173-14-01-01-91] Error deserializating SOAP response body; <<io_obj p:0x000000043fd8f3c8, h:22, <TCP '127.0.0.1 : 14136'>, <TCP '127.0.0.1 : 8307'>>, /sdk>, method: waitForUpdates, e:
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> Error returned by expat parser: not well-formed (invalid token)
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: -->
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> while parsing serialized value of type string
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> at line 19, column 1246634
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: -->
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> while parsing property "subnqn" of static type string
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: -->
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> while parsing serialized DataObject of type vim.host.NvmeController
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> at line 19, column 1246531
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: -->
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> while parsing property "connectedController" of static type ArrayOfHostNvmeController
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: -->
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> while parsing serialized DataObject of type vim.host.NvmeTopology.Interface
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> at line 19, column 1246423
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: -->
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> while parsing property "adapter" of static type ArrayOfHostNvmeTopologyInterface
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: -->
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> while parsing serialized DataObject of type vim.host.NvmeTopology
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> at line 19, column 1244321
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: -->
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> while parsing property "nvmeTopology" of static type HostNvmeTopology
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: -->
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> while parsing serialized DataObject of type vim.host.StorageDeviceInfo
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> error parsing Any with xsiType HostStorageDeviceInfo
YYYY-MM-DDThh:mm:ss.msZ Er(163) Vpxa[2101212]: --> at line 19, column 1190653

localcli nvme controller list

Name                                                                          Controller Number  Adapter  Transport Type  Is Online  Is VVOL
----------------------------------------------------------------------------  -----------------  -------  --------------  ---------  -------
nqn.2019-10.com.kioxia:▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒:XXX0A2MDXXXX               258  vmhba0   PCIe            true       false

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

The NVME controller name is generated with Model Number and EUI64. Sometimes due to device issues, the name contains special characters which results in vpxa agent service crash.

Resolution

Please engage your Hardware vendor to reseat the device to see if that resolves the issue if not replace the device to fix the issue. 

Powered by Wolken Software