VCF Operations deployment fails with Error: LCMVROPSSYSTEM29002 Operations collector group operation failure

Products

VCF Operations

Issue/Introduction

After VCF Operations is deployed by the VCF Installer, the task "Retrieve the status for VCF Operations with VCF Operations collector Deployment request" fails.

Error message displayed in the VCF Installer task screen:

Injecting task failure event. Error Code : 'LCMVROPSSYSTEM29002', Retry : 'true', Causing Properties : '
...
com.vmware.vrealize.lcm.plugin.common.vrops.exceptions.CollectorGroupOperationsException: Wait timeout, Cloud Proxy node didn't join the Operations cluster, retry after sometime.

Check the status of haproxy with the command: systemctl status haproxy.service -l

The startup of haproxy times out.

Aug 15 05:37:27 ###.###.###.### systemd[1]: Starting HAProxy Load Balancer...
Aug 15 05:38:05 ###.###.###.### haproxy[7551]: [NOTICE]   (7551) : config : [/etc/haproxy/haproxy.cfg:87] : 'server PrxyRC_CRUSH_FTP_DEV_BE/CRUSH_FTP_DEV_0' : could not resolve address 'api-devlvn.broadcom.net', disabling server.
Aug 15 05:38:45 ###.###.###.### haproxy[7551]: [NOTICE]   (7551) : config : [/etc/haproxy/haproxy.cfg:96] : 'server PrxyRC_CRUSH_FTP_STAGE_BE/CRUSH_FTP_STG_0' : could not resolve address 'eapi-gcpstg.broadcom.com', disabling server.
Aug 15 05:38:57 ###.###.###.### systemd[1]: haproxy.service: start operation timed out. Terminating.
Aug 15 05:38:57 ###.###.###.### systemd[1]: haproxy.service: Failed with result 'timeout'.
Aug 15 05:38:57 ###.###.###.### systemd[1]: Failed to start HAProxy Load Balancer.
Aug 15 05:38:58 ###.###.###.### systemd[1]: haproxy.service: Scheduled restart job, restart counter is at 1.
Aug 15 05:38:58 ###.###.###.### systemd[1]: Stopped HAProxy Load Balancer.
Aug 15 05:38:58 ###.###.###.### systemd[1]: Starting HAProxy Load Balancer...
Aug 15 05:39:37 ###.###.###.### haproxy[7594]: [NOTICE]   (7594) : config : [/etc/haproxy/haproxy.cfg:87] : 'server PrxyRC_CRUSH_FTP_DEV_BE/CRUSH_FTP_DEV_0' : could not resolve address 'api-devlvn.broadcom.net', disabling server.
Aug 15 05:39:57 ###.###.###.### systemd[1]: haproxy.service: Deactivated successfully.
Aug 15 05:39:57 ###.###.###.### systemd[1]: Stopped HAProxy Load Balancer.

The dig command shows "timed out."

# dig api-devlvn.broadcom.net
;; communications error to ###.###.###.####53: timed out 

; <<>> DiG 9.20.0 <<>> api-devlvn.broadcom.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 65000
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;api-devlvn.broadcom.net.       IN      A

;; Query time: 4416 msec
;; SERVER: ###.###.###.####53(###.###.###.###) (UDP)
;; WHEN: Mon Sep 01 06:33:02 UTC 2025
;; MSG SIZE  rcvd: 52

Environment

VCF Operations Collector 9.x

Cause

When haproxy starts, it attempts DNS resolution for the FQDNs specified in its configuration.

api-devlvn.broadcom.net
eapi-gcpstg.broadcom.com
eapi.broadcom.com

Normally, DNS should quickly return an NXDOMAIN response (even with recursive DNS enabled). However, in this situation, the DNS resolution instead timed out. This indicates an issue with DNS infrastructure, network firewall configuration, or overall DNS responsiveness, rather than recursion itself.

Resolution

Disable DNS recursion so that the DNS server immediately returns a response, preventing upstream query timeouts.

Below is an example configuration using Windows Server DNS.
Note: This is provided as an example only. Before making any DNS configuration changes, always consult with the software vendor providing the DNS service.

Steps:

Access the Windows Server
Open DNS Manager
Right-click the server icon
Select Properties > Advanced
Check the "Disable recursion (also disables forwarders)" option, then click Apply > OK
Right-click the server icon again
Go to All Tasks > Restart

Alternatively workaround the issue by:

ssh to the VCF Operations Collector Appliance using the root account.
Bypass external DNS resolution by modifying the /etc/hosts file to include entries for api-devlvn.broadcom.net, eapi-gcpstg.broadcom.com, eapi.broadcom.com .
Restart systemd-resolved with the command: systemctl restart systemd-resolved
Retry the deployment task from the VCF Installer.