Adding an ESXI Host to vCenter fails with an Error "vapi.invalid.result.code <503 Service Unavailable>" after management IP change

Products

• VMware vCenter Server

Issue/Introduction

• When attempting to add an ESXi host back to vCenter Server inventory (typically after a management IP address change or host removal), the task fails with the following errors:

  • vSphere Client UI:
    
    A failure occurred when starting a host compliance check operation on host <ESXi_Hostname> : Error: system_error Messages: vapi.invalid.result.code<Recv of frame failed with code: 503 Service Unavailable>
  • /var/log/vmware/vpxd/vpxd.log:
    
    YYYY-MM-DDTHH:mm:ss.SSSZ info vpxd[38185] [Originator@6876 sub=vmomi.soapStub[8] opID=mo1m5okn-1263882-auto-r37x-h5:70153921-7d-01-01-LicenseClientUnregisterHostAsync-7
862abcc] SOAP request returned HTTP failure; <<cs p:00007efca80022c0, TCP:localhost:<ID>>, /ls/sdk>, method: unregisterEntity; code: 500(Internal Server Error); fault:
(vim.fault.LicenseEntityNotFound) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    entityId = "host-<ID>"
-->    msg = "Received SOAP response fault from [<<cs p:00007efca80022c0, TCP:localhost:1080>, /ls/sdk>]: unregisterEntity
--> "
--> }

YYYY-MM-DDTHH:mm:ss.SSSZ error vpxd[19679] [Originator@6876 sub=vpxLro opID=mo1m5okn-1263882-auto-r37x-h5:70153921-7d-01-01] [VpxLRO] Unhandled exception: N4Vpxd7Clients29BadVapiConfigurationExceptionE(Error:
-->    system_error
--> Messages:
-->    vapi.invalid.result.code<Recv of frame failed with code: 503 Service Unavailable>
--> )
--> [context]zKq7AVECAQAAALa6fwEbdnB4ZAAAyzdIbGlidm1hY29yZS5zbwAArDA3AA9VOIFI74kCdnB4ZACBrVGKAoFsXYoCgeFfigKBy2CKAoGJBSkCgQkOKQKBtSIpAoHr1kIBgbzxQgGB3PlKAoEtKUsCgSY1LgGBxDYuAYFTay4BgaEArgKBURGuAoHrHq0Cgfb5rQIABPwsAA9VLQDrKkYCxJAIbGliYy5zby42AAKskRA=[/context]
  • /var/log/vmware/cis-license/license.log:
    
    YYYY-MM-DDTHH:mm:ss.SSSZ  invocation-vmomi-executor-1 opId=mo1m5okn-1263882-auto-r37x-h5:70153921-7d-01-01-LicenseClientUnregisterHostAsync-7862abcc  ERROR server.core.oldmanagement.vmomi.LicenseAssignmentManagerMoImpl Missing license entity host-<ID>:0d605cfd-2a60-47b7-8e67-bfb9dcc60b23
com.vmware.cis.license.vpx.server.core.oldmanagement.LicenseAssignmentManagerException$NotFound: Missing license entity host-<ID>:0d605cfd-2a60-47b7-8e67-bfb9dcc60b23
        at com.vmware.cis.license.vpx.server.core.oldmanagement.impl.LicenseAssignmentManagerBaseImpl.queryDaoLicenseEntityOrThrowIfNotFound(LicenseAssignmentManagerBaseImpl.java:438) ~[license-vpx-server-core-9.0.2.jar:?]
        at com.vmware.cis.license.vpx.server.core.oldmanagement.impl.LicenseAssignmentManagerImpl.unregisterEntityInt(LicenseAssignmentManagerImpl.java:1192) ~[license-vpx-server-core-9.0.2.jar:?]

Environment

• VMware vCenter Server 

Cause

This issue occurs because stale registration data or orphaned license entities remain in the vCenter Server database (VCDB) or the VMware Directory Service (vmdir) for the host's previous registration. When vCenter attempts to add the host, the VAPI service encounters a conflict with these existing but "missing" entities, resulting in a 503 Service Unavailable response during the compliance check.

Resolution

To resolve this issue, you must manually remove the stale host and license entries from the vCenter database and the License Service.

Note: Take a file-based backup or snapshot of the vCenter Server Appliance (VCSA) before proceeding.

1. Log in to the VCSA via SSH as root.
2. Search for the stale license entity in the LDAP directory: Review this command before running it.
   

   /usr/bin/ldapsearch -LLL -b "dc=vsphere,dc=local" -s sub -D "cn=administrator,cn=users,dc=vsphere,dc=local" -W > ldif-export.ldif

3. Locate the entry starting with dn: cn=AssetEntity_host-<ID>... that matches the hostname of the impacted host
4. Delete the stale entry: This command will make changes to your system. Review it carefully before running.
   
   ldapdelete -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -W "cn=AssetEntity_host-<ID>-<UUID>,cn=LicenseService,cn=services,dc=vsphere,dc=local"
5. Connect to the vCenter Database:
   
   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
6. Find the ID of the stale host:
7.  Manually remove an ESXi host from the vCenter Server database using KB 314599
8.  Re-add the Host
9. In the vSphere Client, attempt to add the ESXi host to the inventory again using the new IP address or FQDN and it should add successfully