VCF Operations Cloud Proxy Offline

Products

• VCF Operations

Issue/Introduction

• Cloud Proxy is reporting as 'Offline' in the Cloud Proxies menu.
• The same cloud proxy reports its Health Status in the VCP Operations Admin UI as 'Outdated life status'
• The console for the Cloud Proxy in vCenter, as the error below;

  • [FAILED] Failed to start HAProxy Load Balancer.

• On the affected Cloud Proxy VM, the haproxy service fails to start and remains in an activating state when checked using:
  service haproxy status
  
  
• journalctl [journalctl -xe -u haproxy] logs on Cloud Proxy VM shows errors indicating hostname resolution failures:
  
  MM DD hh:mm:ss <Cloud Proxy VM Hostname> haproxy[49287]: [NOTICE] : 'server PrxyRC_BE/VROPS_0' : could not resolve address '<Operations node FQDN>', disabling server.
MM DD hh:mm:ss <Cloud Proxy VM Hostname> haproxy[49287]: [NOTICE] : 'server PrxyRC_INSECURE_BE/VROPS_0' : could not resolve address '<Operations node FQDN>', disabling server.
MM DD hh:mm:ss <Cloud Proxy VM Hostname> systemd[1]: haproxy.service: start operation timed out. Terminating.
MM DD hh:mm:ss <Cloud Proxy VM Hostname> systemd[1]: haproxy.service: Failed with result 'timeout'.


• DNS resolution test from Cloud Proxy VM to the Operations node fails:
  
  root@<CP VM Hostname> [ /var]# nslookup <Operations Node FQDN>
;; communications error to <DNS Server>#53: timed out
;; communications error to <DNS Server>#53: timed out 
;; communications error to <DNS Server>#53: timed out 
;; no servers could be reached

Environment

VCF Operations 9.x

Cause

When haproxy starts, it attempts DNS resolution for the FQDNs listed in /etc/haproxy/haproxy.cfg.
FQDNs which do not exist in the DNS records lead to recursive queries being sent to the upstream DNS server, causing a timeout. This led to the haproxy startup process failing.
The issue may also occur due to an incorrect DNS entry within the Cloud Proxy VM's /etc/resolv.conf file.

Resolution

To resolve this issue, follow the below steps:

• SSH into the Cloud Proxy appliance and update the /etc/resolv.conf file with the correct DNS entry.
• Start the ' haproxy.service' using below command
  

  systemctl start haproxy.service

• Reboot the Cloud Proxy VM if restart service doesn't fix the issue.
• If the issue still persists, please contact Broadcom Support for assistance.