vCenter UI returns "no healthy upstream" error for remote VPN users over conflicting subnets
search cancel

vCenter UI returns "no healthy upstream" error for remote VPN users over conflicting subnets

book

Article ID: 446856

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Remote users establishing a corporate VPN connection receive a "no healthy upstream" and/or "HTTP 503" error when attempting to access the vCenter Server Appliance (VCSA) vSphere Client via its Fully Qualified Domain Name (FQDN) or direct IP address.

The environment exhibits the following symptoms and network-layer behaviors:

  • Running nslookup against the vCenter FQDN from the problematic remote site successfully resolves to the destination IP address, matching the local network resolution output exactly such as resolving uniformly to an IP like 192.##.##.250.

  • One or more users connecting concurrently from alternative remote VPN sites and the internal local network can access and log into the vCenter UI normally during this time.

  • Network-layer reachability tests via ping are completely successful with 0% packet loss in both directions between the remote workstation and the vCenter appliance.

  • The remote connection utilizes a split-tunnel VPN client (No Default Gateway), and the IP address assigned for the local LAN/Wi-Fi is in the same subnet as the vCenter local address.

    C:\> ipconfig /all

    Windows IP Configuration

    Ethernet adapter Ethernet <#>:

       Connection-specific DNS Suffix  . : <Corporate.domain.local>           
       Description . . . . . . . . . . . : Virtual Network Adapter
       Physical Address. . . . . . . . . : 00-##-##.##-##-55
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes
       Link-local IPv6 Address . . . . . : fe80::####:####::####::####:%14(Preferred)
       IPv4 Address. . . . . . . . . . . : 10.##.##.2(Preferred)
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
          Default Gateway . . . . . . . . . :                           <-- [Blank By Design ]

    Wireless LAN adapter Wi-Fi:

       Connection-specific DNS Suffix  . : <Local_Network>
       Description . . . . . . . . . . . : #### Wi-Fi 6E #### 160MHz
       Physical Address. . . . . . . . . : 00-15-5D-A1-B2-C3
       DHCP Enabled. . . . . . . . . . . : Yes
       Autoconfiguration Enabled . . . . : Yes
       IPv4 Address. . . . . . . . . . . : 192.168.1.220(Preferred)  <-- [Wi-Fi/LAN IP]
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . : 192.##.##.1               <-- [Local Router IP]

  • Consecutive path traces may show a completely clean path to the destination, as the underlying routing conflict may only trigger under heavy multiplexed browser loads rather than sequential diagnostic traffic, causing the trace output to oscillate between a symmetric path and an overlapping path as shown below:

    C:\> tracert -d 192.##.##.250
    Tracing route to 192.##.##.250 over a maximum of 30 hops:

      1     2 ms     2 ms     3 ms  10.##.##.1             <-- [VPN GATEWAY INGRESS]
      2    10 ms     9 ms    11 ms  10.##.##.254           <-- [INTERNAL TRANSIT ROUTER]
      3    12 ms    12 ms    13 ms  192.##.##.250          <-- [vCENTER TARGET]

    Trace complete.

    C:\> tracert -d 192.##.##.250
    Tracing route to 192.##.##.250 over a maximum of 30 hops:

      1    <1 ms    <1 ms    <1 ms  192.##.##.1            <-- [LOCAL GATEWAY INTERFERENCE]
      2     5 ms     4 ms     5 ms  10.##.##.1             <-- [VPN GATEWAY INGRESS]
      3    12 ms    12 ms    11 ms  10.##.##.254           <-- [INTERNAL TRANSIT ROUTER]
      4    14 ms    13 ms    14 ms  192.##.##.250          <-- [vCENTER TARGET]

    Trace complete.

  • The vCenter Envoy proxy access logs (/var/log/vmware/envoy/envoy-access.log) record stream drop timeouts (UT) and upstream routing failures (UH):

    [<DATE>T06:55:12.143Z] "GET /ui/ HTTP/2" 503 UT 0 0 15000 - "<CLIENT_VPN_IP_192.##.##.220>" "Mozilla/5.0..." "<vCenter_FQDN>" "127.0.0.1:443"
    [<DATE>T06:55:12.185Z] "POST /ui/api/v1/session HTTP/2" 503 UH 0 0 0 - "<CLIENT_VPN_IP_192.##.##.220>" "Mozilla/5.0..." "<vCenter_FQDN>" "-"
    [<DATE>T06:55:13.010Z] "GET /ui/static/css/main.css HTTP/2" 503 UT 0 0 15000 - "<CLIENT_VPN_IP_192.##.##.220>" "Mozilla/5.0..." "<vCenter_FQDN>" "127.0.0.1:443"

Environment

VMware vCenter Server

Cause

  • As a default industry standard, split-tunnel VPN configurations omit a default gateway on the virtual adapter, relying solely on injected static routes to steer corporate traffic down the tunnel.

  • While a highly specific host route (such as a /32 binding) successfully forces stateless ICMP traffic (ping and tracert) down the secure virtual path, an address space overlap between the remote users endpoint local subnet (192.##.##.0/24) and the corporate vCenter management subnet (192.##.##.0/24) across a split-tunnel VPN connection triggers metric priority conflicts on the guest operating system.
  • Interface metric conflicts cause multiplexed HTTP/2 streams to leak via the physical gateway rather than remaining encapsulated in the virtual VPN tunnel, leading to incomplete transport streams, hardcoded timeout limits, and local HTTP 503 UT or UH errors. 

Resolution

For Overlapping Subnet Deployments:

  • Adjust the Guest OS network interface metrics to assign a lower metric cost (highest routing priority) to the virtual VPN adapter.

  • Reconfigure the remote home network router's DHCP server subnet pool to utilize an alternate, non-overlapping private IP network block (such as a 172.16.x.x segment).

    • Note: This action also resolves instances where a standard home gateway IP (e.g., 192.##.XX.1) conflicts directly with core corporate infrastructure assets.

For Non-Overlapping Subnet Deployments (Or if the issue persists even after the above actions):

  • Disconnect the user workstation from the current Wi-Fi/LAN network and route connection traffic through a different internet hotspot, e.g., a mobile hotspot, to eliminate potential local router or ISP-specific issues such as firewall blocks, tunnel MTU overhead constraints, etc.

  • Lower the Maximum Transmission Unit (MTU) size on the client workstation's virtual VPN network adapter interface to 1350 bytes.

If the different internet hotspot connection works successfully but lowering the MTU size on the home network does not resolve the issue, please contact your local internet service provider (ISP) or internal home router administrator to address upstream network blockages or hardware MTU restrictions

Additional Information

"no healthy upstream" error.

Error "503 Service Unavailable".

VPN client in Split Tunnel mode.

IPsec tunnel overhead and encryption overhead will significantly reduce the available payload space.

Packets that have a do not fragment bit set to 1 will not be fragmented and will be discarded if they exceed the maximum size.

 

Powered by Wolken Software