VCF Operations Inventory Sync Fails with IllegalStateException: Cannot determine IP address for FQDN

Products

• VCF Operations

Issue/Introduction

• When attempting to perform an inventory sync in VMware Cloud Foundation (VCF) Operations—specifically after adding an ESXi host out-of-band (OOB) to a brownfield or imported workload domain—the task fails.
• The following error patterns are observed in /var/log/vmware/vcf/domain-manager/domainmanager.log:

YYYY-MM-DDthh:mm:sec.319+0000 WARN  [vcf_dm,6a32b3f11a89509028df############,f223] [c.v.v.discovery.utils.VcenterHelper,dm-exec-1792]  Couldn't find vcenter VM by hostname 10.#.#.# in vc [sddc_FQDN]
YYYY-MM-DDthh:mm:sec..319+0000 ERROR [vcf_dm,6a32b3f11a89509028df############,f223] [c.v.e.s.common.util.NetworkService,dm-exec-1792]  Cannot determine IP address for FQDN 10.#.#.# Invalid IP address 10.#.#.# found while resolving FQDN 10.#.#.#
YYYY-MM-DDthh:mm:sec.319+0000 WARN  [vcf_dm,6a32b3f11a89509028df############,f223] [c.v.v.discovery.utils.VcenterHelper,dm-exec-1792]  Failed to 'Resolve hostname 10.#.#.#.#' due to exception
java.lang.IllegalStateException: Cannot determine IP address for FQDN 10.#.#.# Invalid IP address 10.#.#.# found while resolving FQDN 10.#.#.#
        at com.vmware.evo.sddc.common.util.NetworkService.validateResolvedIp(NetworkService.java:714)
      cManagerBase,dm-exec-1792]  Finding vms with host name 10.#.#.#
YYYY-MM-DDthh:mm:sec..319+0000 WARN  [vcf_dm,6a32b3f11a89509028df##########,f223] [c.v.v.discovery.utils.VcenterHelper,dm-exec-1792]  Failed to 'Find VM by hostname 10.#.#.# in vCenter [sddc_FQDN]' due to exception
com.vmware.vim.binding.vmodl.fault.SystemError: null
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.checkNonOptionalValueProvided(ResponseImpl.java:257)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:237)

Environment

• VMware Cloud Foundation 9.x

Cause

The issue is caused by a failure in reverse DNS (Domain Name System) resolution. Specifically, a missing or incorrect PTR (Pointer) record for a management component (such as the SDDC Manager) prevents the VCF deployment engine from performing the necessary lookups required to validate and sync the environment's inventory.

Resolution

Ensure that both forward and reverse DNS records are correctly configured for all VCF management components.

1. Verify the DNS record.
   
   • Log in to DNS server.
   • Verify the Forward (A) and Reverse (PTR) records for the SDDC Manager, vCenter Server, and the newly added ESXi hosts.
   • Correct any invalid or missing PTR records to ensure they point to the correct FQDN.
2. Verify Resolution:
   • From the SDDC Manager or a management VM, verify resolution using nslookup:
     • nslookup [FQDN]
     • nslookup [IP_ADDRESS]
3. Re-sync Inventory
   
   • Log in to the VCF Operations UI.
   • Navigate to Fleet Management > Lifecycle > VCF Management > Components.
   • Select sync inventory.