Load balancer health monitor loses connectivity when the health-check port stops accepting HTTP in VCF Automation

Products

• VCF Automation

Issue/Introduction

• After upgrading VCF Automation to 9.1.x (or on a fresh 9.x install) behind an external load balancer, the provider and/or tenant portal UI becomes inaccessible.
  
  
• VCF Operations registration step in the VCFA automation ugprade workflow may fail to connect to Automation FQDN.

• The health-reporter-app container logs show:

  http: TLS handshake error from <load balancer IP>: client sent an HTTP request to an HTTPS server

• The health endpoint no longer responds over HTTP, confirm with curl

curl http://<VCFA_FQDN>:8008/health

• Expected response:
  
  Client sent an HTTP request to an HTTPS server.

Environment

VCF Automation 9.1 

Cause

Starting with VCF Automation 9.1.0, the health-check port (8008) was migrated to TLS:

• On greenfield VCF Automation 9.1.0 and later installations, port 8008 serves /health over HTTPS only.
• On environments upgraded from vRealize Automation 8 or VCF Automation 9.0.x, port 8008 continues to accept HTTP traffic, and HTTP support stays enabled as long as HTTP traffic keeps arriving from the existing load balancer.

HTTP access on port 8008 is automatically disabled if traffic stops for a continuous period, specifically 3 hours in version 9.1.0 and 12 hours in versions 9.1.1 and later. This inactivity typically occurs when a load balancer's health monitor is temporarily paused or removed. Consequently, if the load balancer is still configured for HTTP checks after this timeout triggers, it will fail to verify node health when it attempts to resume sending traffic.

 

Resolution

Reconfigure the load balancer to probe health over HTTPS instead of HTTP.

Resolution:

1. On the external load balancer, edit the active health monitor for the VCF Automation server pool.
2. Change the protocol to HTTPS, keep the port at 8008.
3. Set the request URL to /health exactly.
4. If you need the ingress certificate, retrieve it from the VCF Services Runtime (VMSP) management API:

RUNTIME_FQDN="<VCF Services Runtime FQDN>"
PASSWORD='<vmware-system-user password>'

TOKEN=$(curl -sk --request POST \
  --url "https://${RUNTIME_FQDN}/api/v1/identity/token" \
  --header 'content-type: application/x-www-form-urlencoded' \
  --data grant_type=password \
  --data [email protected] \
  --data "password=${PASSWORD}" | jq -r .access_token)

VCFA_ID=$(curl -sk "https://${RUNTIME_FQDN}/api/v1/components" \
  --header "Authorization: Bearer ${TOKEN}" | jq -r '.components[] | select(.type == "vcfa") | .id')

curl -sk "https://${RUNTIME_FQDN}/api/v1/components/${VCFA_ID}" \
  --header "Authorization: Bearer ${TOKEN}" \
  | jq -r '.spec.configuration.ingress.vcfa.tls.ca' > vcfa-ingress-ca.pem

      5. Confirm the pool reports healthy and both portals load correctly.

Applies to: VCF Automation 9.1.0 and later, in any deployment with an external/third-party load balancer in front of the native ingress.