Running VCF installer fails at "Prepare input for adding VCF Operations collector node to existing VCF Operations" step.

Products

• VMware Cloud Foundation

Issue/Introduction

• When running the VMware Cloud Foundation (VCF) installer, the deployment fails at the "Prepare input for adding VCF Operations collector node to existing VCF Operations" step. The following error is displayed in the user interface:
  
  Environment details retrieval for the given hostname Failed <VCF Operations FQDN>.
  
  
• In the SDDC Manager log file /var/log/vmware/vcf/domainmanager/domainmanager.log, IP address of VCF Operations is returned in the "address" field instead of the expected FQDN:
  
  Received VCF Operations cluster status: {"nodes_states":[{"name":"<node name>","address":"<ip address>","uuid":"<uuid>","role":"MASTER","state":"CONFIGURED"}],"cluster_state":"INITIALIZED"}"
......
GET_ENVIRONMENT_BY_HOSTNAME_FAILED Environment details retrieval for the given hostname failed <VCF Operations FQDN>
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Environment details retrieval for the given hostname failed <VCF Operations FQDN>
  
  
• When logging into the VCF Operations Admin UI, an IP address is displayed under "Node Address" field instead of FQDN.
• Both forward and reverse DNS records for VCF Operations are verified to be correct and resolvable with below commands:
  
  nslookup <VCF Operations FQDN>
nslookup <VCF Operations IP address>

Environment

VMware Cloud Foundation 9.0.x

Cause

This issue occurs if VCF Operations was initially deployed using its IP address instead of its FQDN. This typically happens if DNS resolution fails during the initial bring-up process.

The VCF installer specifically relies on the VCF Operations FQDN to query and retrieve environment details from the Fleet Management appliance. The VCF installer begins by fetching all environment records from Fleet Management appliance and looks for a match against the VCF Operations FQDN. If there is a match, VCF installer uses that environment and adds VCF Operations collector to it.

If the VCF Operations was originally deployed using IP address, the environment details will be populated with the IP address instead of FQDN, which will prevent VCF installer from retrieving the correct VCF Operations environment details.

Resolution

As a workaround, follow below steps to manually inject the FQDN entry into the Fleet Management appliance inventory database:

1. Log in to the Fleet Management appliance via SSH using the root account.
2. Switch to the postgres user and connect to the database instance by executing:
   
   su - postgres
/opt/vmware/vpostgres/current/bin/psql -U postgres vrlcm
   
   
3. Run the following query to locate the "node_vmid" associated with your master node. Note this value down, as it will be required for the insertion step:
   
   select * from vm_lcops_node_properties;
   
   
4. Insert a new entry into VCF Operations Fleet Management database with VCF Operations FQDN.
   Replace <node_vmid> with the actual "node_vmid" obtained from step 3 and <FQDN> with the actual FQDN of VCF Operations.
   
   insert into vm_lcops_node_properties (node_vmid, value, key)
values ('<node_vmid>', '<FQDN>', 'hostName');

Retry the failed step in VCF installer after applying the workaround.