Error "A general system error occurred host management agents not reachable" when adding host to vCenter

Products

• VMware vSphere ESXi

Issue/Introduction

• vCenter logs (/var/log/vmware/vpxd/vpxd,log) indicate existing users and policy applied/exists 

2025-08-26T03:17:23.447Z info vpxd[06373] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=5f2770bf] [PolicyUpdater] Grants for client profile LU###vpxuser on host <Host FQDN> are already in desired state
2025-08-26T03:17:23.450Z warning vpxd[06373] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=5f2770bf] [CreateOrUpdate] Policy with external group name TrustedAdmins already exists on host <Host FQDN>. Exception: N4Vpxd7Clients22AlreadyExistsExceptionE(Error:
-->    com.vmware.vapi.std.errors.already_exists

2025-08-26T03:17:23.454Z info vpxd[06373] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=5f2770bf] [PolicyUpdater] Grants for client profile SG#https://<vCenter FQDN>/websso/SAML2/Metadata/vsphere.local#vsphere.local#TrustedAdmins on host <Host FQDN> are already in desired state
2025-08-26T03:17:23.456Z warning vpxd[06373] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=5f2770bf] [CreateOrUpdate] Policy with external group name vStatsGroup already exists on host <Host FQDN>. Exception: N4Vpxd7Clients22AlreadyExistsExceptionE(Error:
-->    com.vmware.vapi.std.errors.already_exists
--> No messages!

• vpxa agent logs (/var/run/log/vpxa.log) on ESXi indicates backtrace when nvme device being queried and resulting in the service crashing down. 

2025-08-26T07:13:22.233Z Er(163) Vpxa[2101241]: [Originator@6876 sub=vmomi.soapStub[5] opID=mats8od4-250817-auto-5dj8-h5:70029173-14-01-01-91] Error deserializating SOAP response body; <<io_obj p:0x000000043fd8f3
c8, h:22, <TCP '127.0.0.1 : 14136'>, <TCP '127.0.0.1 : 8307'>>, /sdk>, method: waitForUpdates, e:
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> Error returned by expat parser: not well-formed (invalid token)
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: -->
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> while parsing serialized value of type string
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> at line 19, column 1246634
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: -->
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> while parsing property "subnqn" of static type string
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: -->
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> while parsing serialized DataObject of type vim.host.NvmeController
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> at line 19, column 1246531
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: -->
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> while parsing property "connectedController" of static type ArrayOfHostNvmeController
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: -->
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> while parsing serialized DataObject of type vim.host.NvmeTopology.Interface
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> at line 19, column 1246423
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: -->
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> while parsing property "adapter" of static type ArrayOfHostNvmeTopologyInterface
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: -->
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> while parsing serialized DataObject of type vim.host.NvmeTopology
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> at line 19, column 1244321
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: -->
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> while parsing property "nvmeTopology" of static type HostNvmeTopology
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: -->
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> while parsing serialized DataObject of type vim.host.StorageDeviceInfo
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> error parsing Any with xsiType HostStorageDeviceInfo
2025-08-26T07:13:22.233Z Er(163) Vpxa[2101212]: --> at line 19, column 1190653

localcli nvme controller list 
Name                                                                          Controller Number  Adapter  Transport Type  Is Online  Is VVOL
----------------------------------------------------------------------------  -----------------  -------  --------------  ---------  -------
nqn.2019-10.com.kioxia:▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒:XXX0A2MDXXXX               258  vmhba0   PCIe            true       false

Environment

ESXi 7.x

ESXi 8.x

Cause

The NVME controller name is generated with Model Number and EUI64. Sometimes due to device issues, the name contains special characters which results in vpxa agent service crash.

Resolution

Please engage your Hardware vendor to reseat the device to see if that resolves the issue if not replace the device to fix the issue.