The firstboot of the VCF Operations Collector (Cloud Proxy) fails because haproxy does not start and times out.

Products

• VCF Operations

Issue/Introduction

• After VCF Operations is deployed by the VCF Installer, the task "Retrieve the status for VCF Operations with VCF Operations collector Deployment request" fails.
• Error message displayed in the VCF Installer task screen:
  

  Injecting task failure event. Error Code : 'LCMVROPSSYSTEM29002', Retry : 'true', Causing Properties : '
  ...
  com.vmware.vrealize.lcm.plugin.common.vrops.exceptions.CollectorGroupOperationsException: Wait timeout, Cloud Proxy node didn't join the Operations cluster, retry after sometime.

   

• The startup of haproxy times out.
  

  Aug 15 05:37:27 XXX.XXX.XXX.XXX systemd[1]: Starting HAProxy Load Balancer...
  Aug 15 05:38:05 XXX.XXX.XXX.XXX haproxy[7551]: [NOTICE]   (7551) : config : [/etc/haproxy/haproxy.cfg:87] : 'server PrxyRC_CRUSH_FTP_DEV_BE/CRUSH_FTP_DEV_0' : could not resolve address 'api-devlvn.broadcom.net', disabling server.
  Aug 15 05:38:45 XXX.XXX.XXX.XXX haproxy[7551]: [NOTICE]   (7551) : config : [/etc/haproxy/haproxy.cfg:96] : 'server PrxyRC_CRUSH_FTP_STAGE_BE/CRUSH_FTP_STG_0' : could not resolve address 'eapi-gcpstg.broadcom.com', disabling server.
  Aug 15 05:38:57 XXX.XXX.XXX.XXX systemd[1]: haproxy.service: start operation timed out. Terminating.
  Aug 15 05:38:57 XXX.XXX.XXX.XXX systemd[1]: haproxy.service: Failed with result 'timeout'.
  Aug 15 05:38:57 XXX.XXX.XXX.XXX systemd[1]: Failed to start HAProxy Load Balancer.
  Aug 15 05:38:58 XXX.XXX.XXX.XXX systemd[1]: haproxy.service: Scheduled restart job, restart counter is at 1.
  Aug 15 05:38:58 XXX.XXX.XXX.XXX systemd[1]: Stopped HAProxy Load Balancer.
  Aug 15 05:38:58 XXX.XXX.XXX.XXX systemd[1]: Starting HAProxy Load Balancer...
  Aug 15 05:39:37 XXX.XXX.XXX.XXX haproxy[7594]: [NOTICE]   (7594) : config : [/etc/haproxy/haproxy.cfg:87] : 'server PrxyRC_CRUSH_FTP_DEV_BE/CRUSH_FTP_DEV_0' : could not resolve address 'api-devlvn.broadcom.net', disabling server.
  Aug 15 05:39:57 XXX.XXX.XXX.XXX systemd[1]: haproxy.service: Deactivated successfully.
  Aug 15 05:39:57 XXX.XXX.XXX.XXX systemd[1]: Stopped HAProxy Load Balancer.

   

• The dig command shows "timed out."
  

  # dig api-devlvn.broadcom.net
  ;; communications error to XXX.XXX.XXX.XXX#53: timed out 
  
  ; <<>> DiG 9.20.0 <<>> api-devlvn.broadcom.net
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 65000
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
  
  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 4000
  ;; QUESTION SECTION:
  ;api-devlvn.broadcom.net.       IN      A
  
  ;; Query time: 4416 msec
  ;; SERVER: XXX.XXX.XXX.XXX#53(XXX.XXX.XXX.XXX) (UDP)
  ;; WHEN: Mon Sep 01 06:33:02 UTC 2025
  ;; MSG SIZE  rcvd: 52

Environment

VCF Operations Collector 9.x

Cause

When haproxy starts, it attempts DNS resolution for the FQDNs listed in /etc/haproxy/haproxy.cfg.

In this case, the customer’s environment attempted to resolve the following two hostnames, which did not exist in the DNS records. As a result, recursive queries were sent to the upstream DNS server, causing a timeout. This led to the haproxy startup process failing.

• api-devlvn.broadcom.net
• eapi-gcpstg.broadcom.com

Resolution

Disable DNS recursion so that the DNS server immediately returns a response, preventing upstream query timeouts.

Below is an example configuration using Windows Server DNS.
Note: This is provided as an example only. Before making any DNS configuration changes, always consult with the software vendor providing the DNS service.

Steps:

1. Access the Windows Server
2. Open DNS Manager
3. Right-click the server icon
4. Select Properties > Advanced
5. Check the "Disable recursion (also disables forwarders)" option, then click Apply > OK
6. Right-click the server icon again
7. Go to All Tasks > Restart