Connect-VIServer proxy tunnel request fails with status code 500 or 503 in VCF Operations Orchestrator

Products

• VCF Operations

Issue/Introduction

When running custom PowerCLI workflows in VCF Operations Orchestrator to manage vCenter Server endpoints, connections fail, and scripts may terminate with the following:

Connect-VIServer The proxy tunnel request to proxy 'http://proxy-service.prelude.svc.cluster.local:3128/' failed with status code '500'.

Additionally, 503 and 200 connection statuses are logged in the Orchestrator proxy service (Squid):

TCP_TUNNEL/503 0 CONNECT vCenterFQDN:443 - HIER_DIRECT/IP-Address -
TCP_TUNNEL/200 6575 CONNECT vCenterFQDN:443 - HIER_DIRECT/IP-Address -
TCP_TUNNEL/503 0 CONNECT vCenterFQDN:443 - HIER_NONE/- -

Environment

• VCF Operations Orchestrator 9.1
• vCenter Server (6.5 to 8.x)

Cause

Connecting to vCenter Server endpoints via IP address instead of a Fully Qualified Domain Name (FQDN) can result in PNID mismatches and WebSSO failures.

In version 9.1, any vCenter endpoint that experiences WebSSO failures triggers an increment in a proxy failure counter for these attempted connections. If the failure counter reaches a threshold of 1,000, the internal Squid proxy automatically blocks communication to protect the environment, resulting in 500 or 503 proxy tunnel errors.

Resolution

To stop the proxy from blocking the connection, the underlying authentication and routing issues must be resolved:

1. Resolve all certificate, DNS, and hostnaming misconfigurations on the target vCenter Server endpoints.

2. Verify that you can log into the vCenter Server UI using the FQDN with a valid certificate and no WebSSO redirection failures with the same user Orchestrator is attempting to communicate with.

3. Update the custom PowerCLI scripts to target the proper FQDNs instead of IP addresses arrays.