VCF Operations Adapter Instance Object is down with error: Agent can't send heartbeats because socket address structure initialization is failing

Products

• VMware vCenter Server
• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware Cloud Foundation
• VMware vSphere ESXi

Issue/Introduction

• VCF Operations / Aria Operations reports an Adapter Instance Object is down alert for ESXi hosts
  
  
• The affected ESXi host periodically disconnects from the vCenter
  
  
• vSphere Client reporting the following error regarding the ESXi host:
  
  Agent can't send heartbeats because socket address structure initialization is failing
  
  
• Hosts logs reporting the following errors:
  

  YYYY-MM-DDThh:mm:ssZ Er(163) Vpxa[2099551]: [Originator@6876 sub=Heartbeat opID=vpxaHeartbeat.cpp:218-680f5dba] getaddrinfo of server address failed; '<vc_fqdn>', r: -2(Name or service not known)

• Executing the following command on the ESXi host on CLI reports the FQDN instead of the IP address of the vCenter:
  
  configstorecli config current get -c esx -g services -k vpxa_solution_user_config | grep 'server_ip'

   "server_ip": "<vc_fqdn>",

  
  

Environment

VMware Cloud Foundation 9.x
VMware Cloud Foundation 5.x
VMware Aria Suite
VMware vCenter 9.x
VMware vCenter 8.x
VMware vSphere ESXi 9.x
VMware vSphere ESXi 8.x

Cause

The configured heartbeat IP in the host configuration (server_ip) is expected to be the an IP address and not an FQDN and should match the configured heartbeat IP address of the vCenter

Resolution

1.  Retrieve the configured heartbeat address of the vCenter Server from the vSphere Client:
   
   vCenter > Configure > Advanced Settings > VirtualCenter.AutoManagedIPV4
   
   
2. Set the IP address using configstorecli command (replace <vcenter_ip> with the correct IP address):
   
   configstorecli config current set -c esx -g services -k vpxa_solution_user_config --path /server_ip --value "<vcenter_ip>"
   
   
3. Verify the heartbeat IP address is correct:
   
   configstorecli config current get -c esx -g services -k vpxa_solution_user_config --path /server_ip
   
   
4. Restart the vpxa service on the host to apply the change:
   
   /etc/init.d/vpxa restart

Additional Information

The same error can be observed if the IP of the vCenter have been changed:

337654 - Changing vCenter Server IP address causes ESXi hosts to disconnect