Cloud Proxy node periodically loses connectivity to VCF Operations with 503 errors

Products

• VCF Operations

Issue/Introduction

Cloud Proxy node periodically loses connectivity to VMware Cloud Foundation Operations (VCF Ops).

In /var/log/haproxy-traffic.log, the following entries are observed, where the <NOSRV> and SC-- flags indicate that no backend server was available due to the internal DNS resolver failing to resolve the backend hostname:

haproxy[<pid>]: <REDACTED_IPS>:<port> [<date>] PrxyRC_FE PrxyRC_BE/<NOSRV> 1/-1/-1/-1/<time> 503 <bytes> - - SC- <conn>/<conn>/<conn>/<conn>/<retries> 0/0 "POST /casa/... HTTP/1.1"
haproxy[<pid>]: <REDACTED_IPS>:<port> [<date>] PrxyRC_UNSECURE_FE PrxyRC_UNSECURE_BE/<NOSRV> 1/-1/-1/-1/<time> 503 <bytes> - - SC- <conn>/<conn>/<conn>/<conn>/<retries> 0/0 "GET /casa/... HTTP/1.1"

Environment

VCF Operations 9.x 

Cause

By default, the CPRC HAProxy configuration includes a built-in DNS resolver (resolvers resolvernameservers) that parses /etc/resolv.conf, manages its own DNS caching with hold timers and retry logic, and appends resolver directives to all backend server entries. In certain environments this internal resolver fails to correctly resolve or refresh backend hostnames. When that happens, HAProxy cannot route traffic to any backend and returns 503 Service Unavailable. 

Resolution

• Establish an SSH connection to the affected Cloud Proxy node.

• Create the resolver disable flag file by executing the following command:
  
  touch /storage/db/vmware-vrops-cprc/configuration/cprc.haproxy.resolvers.enabled

• Regenerate the HAProxy configuration by executing the provided Python script:
  
  $VMWARE_PYTHON_BIN /usr/lib/vmware-vrops-cprc/scripts/python/cprcHAProxyConfiguration.py

• Restart the HAProxy service to apply the new configuration:
  
  systemctl restart haproxy.service
  
  
• Verify that HAProxy is running and HTTP 200 responses are returned from the primary node.

Additional Information

When the flag file is present, cprcHAProxyConfiguration.py skips the resolvers resolvernameservers section entirely and removes all resolver-specific directives from backend server entries. Hostname resolution is then delegated to the OS system resolver (/etc/resolv.conf via libc).

The flag file resides in a persistent storage path (/storage/db/vmware-vrops-cprc/configuration/) that survives both system reboots and product upgrades.