Cloud Proxy deployment failure with error LCMVROPSSYSTEM29002 in VCF Operations
search cancel

Cloud Proxy deployment failure with error LCMVROPSSYSTEM29002 in VCF Operations

book

Article ID: 438174

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

Deployment and Configuration stages for Cloud Proxy fails during the initialization process in VCF Operations. This issue frequently occurs in "Dark Site" or air-gapped environments where outbound traffic is restricted or DNS resolution is limited.

  • Cloud Proxy deployment fails at Stage 8 with error: LCMVROPSSYSTEM29002 — "Operations collector group operation failure – Wait timeout, Cloud Proxy node didn’t join the Operations cluster".
  • Fleet Management UI status remains "Not Ready".
  • HAProxy service fails to start; systemctl status haproxy shows "activating (start)" or "failed".
  • Logs in /var/log/haproxy-admin.log are missing or service status indicates: option 'accept-invalid-http-request' is deprecated.
  • Configuration errors: could not resolve address 'api-devlvn.broadcom.net', disabling server.

systemctl status haproxy

systemd[1]: Starting HAProxy Load Balancer ...
haproxy [30492]: [NOTICE]
haproxy [30499] : [NOTICE]
haproxy [30499]: [NOTICE]
haproxy [30499]: [WARNING]
haproxy [30499]: [WARNING]
haproxy [30499]: [WARNING]

(30492) : Initializing new worker (30499)
(30499) : haproxy version is 3.2.1-f4dla4e
(30499) : path to executable is /usr/sbin/haproxy
(30499) : config : parsing [/etc/haproxy/haproxy.cfg: 43]: option 'accept-invalid-http-request' is deprecated. please use "option accept-unsafe-violations-in-http-request" if absolutely
(30499) : config : parsing [/etc/haproxy/haproxy.cfg:57]: option 'accept-invalid-http-request' is deprecated. please use "option accept-unsafe-violations-in-http-request" if absolutely

 

haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: enabled)
Active: activating (start) since Day YYYY-MM-DD HH:MM:Sec UTC; 1min 18s ago
Main PID: 8318 (haproxy)
Tasks: 2 (limit: 19136)
Memory: 15.0M
CPU: 97ms
CGroup: /system.slice/haproxy.service
I-8318 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -S /var/run/haproxy-master.sock
-8324 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -S /var/run/haproxy-master.sock

Apr 23 systemd[1]: Starting HAProxy Load Balancer ...
(8318) : Initializing new worker (8324)
(8324) : haproxy version is 3.2.1-f4d1a4e
(8324) : path to executable is /usr/sbin/haproxy
config : parsing [/etc/haproxy/haproxy.cfg:43]: option 'accept-invalid-http-request' is deprecated. please use 'option accept-unsafe-violations-in-http-request' if absolutely needed.
(8324) : config : parsing [/etc/haproxy/haproxy.cfg:57]: option 'accept-invalid-http-request' is deprecated. please use 'option accept-unsafe-violations-in-http-request' if absolutely needed.
[/etc/haproxy/haproxy.cfg:87] : 'server PrxyRC_CRUSH_FTP_DEV_BE/CRUSH_FTP_DEV_0' : could not resolve address 'api-devlvn.broadcom.net', disabling server.
(8324) : [/etc/haproxy/haproxy.cfg:96] : 'server PrxyRC_CRUSH_FTP_STAGE_BE/CRUSH_FTP_STG_0' : could not resolve address 'eapi-gcpstg.broadcom.com', disabling server.

 

 

Environment

  • VMware Cloud Foundation (VCF) Operations 9.0.x

Cause

  • Use of option accept-invalid-http-request which is no longer supported in current versions.
  • SSL Verification: Fatal errors occurring during SSL certificate verification before the cluster join is complete.
  • Unreachable FQDNs: HAProxy service hangs attempting to resolve external Broadcom FQDNs (api-devlvn.broadcom.net, eapi-gcpstg.broadcom.com) in environments with restricted outbound access, leading to a Stage 8 timeout.

Resolution

Follow these steps to address HAProxy deprecation and bypass unreachable external lookups:

  1.  Address HAProxy Deprecation and SSL Verification

    1. SSH into the Cloud Proxy VM as the root user.

    2. Open the HAProxy configuration file for editing: vi /etc/haproxy/haproxy.cfg.

    3. Locate the line option accept-invalid-http-request and comment it out by adding a # at the beginning.

    4. To bypass certificate verification issues during initial join, locate all occurrences of: ssl verify required ca-file /storage/vcops/user/conf/ssl/haproxy.ca.pem and replace them with: ssl verify none
       Note: This is a temporary security bypass and should be reverted back referring article Cloud Proxy shows as Offline after upgrading VMware Aria Operations

  2. Bypass External FQDN Lookups

    1. Edit the hosts file: vi /etc/hosts.

    2. Map unreachable Broadcom addresses to the local loopback to prevent resolution hangs:

      • 127.0.0.1 api-devlvn.broadcom.net

      • 127.0.0.1 eapi-gcpstg.broadcom.com

  3. Verify and Restart

    1. Run haproxy -c -f /etc/haproxy/haproxy.cfg to confirm the configuration is valid.

    2. Restart the service: systemctl restart haproxy

Powered by Wolken Software