"Error downloading plug-in. URL is unreachable. Read timed out" Observed During VMware LCI Deployment.

Products

• VMware NSX
• VMware vSphere Kubernetes Service

Issue/Introduction

• Local Consumption Interface (LCI) deployment is failing with the error:
  "Cannot complete the operation. See the event log for details. Error downloading plug-in. URL is unreachable. Read timed out."
  
  
• vSphere client is reporting a a "Failed" status while trying to register the plugin manifest from the LCI appliance at https://<Supervisor Cluster FIP>:443/appplatform1/plugin-9.0.x.json.
  
  
• The below curl command from vCenter results in a '504 Gateway Timeout':
  curl -v https://<Supervisor Cluster FIP>:443
  
  
• Ping from Supervisor VM to vCenter succeed swith a packet size of 1500 or less:
  ping -M do -s 1472 <vCenter IP>
  
  
• Ping from Supervisor VM to vCenter with a packet size above 1500 fails with error "message too long":
  ping -M do -s 1473 <vCenter IP>
   
• vmkping from Supervisor VM host TEPs to Edge TEPs succeeds with a packet size of 1500 or less:
  vmkping -S vxlan -I vmk10 -d -s 1472 <Edge TEP IP>   
  
  
• vmkping from Supervisor VM host TEPs to Edge TEPs with a packet size above 1500 fails: 
  vmkping -S vxlan -I vmk10 -d -s 1473  <Edge TEP IP>

Environment

VMware NSX

vSphere Kubernetes Service

Cause

The LCI deployment is failing due to an MTU bottleneck on the physical VLAN backing the overlay network.  This is because NSX overlay traffic utilizes Geneve encapsulation which introduces additional packet overhead.

Consequently, packets exceeding 1472 bytes, such as those sent during the TLS certificate exchange are dropped between the Supervisor VM ESX host TEPs and Edge TEPs.

Resolution

This is a condition that may occur in a VMware NSX environment due to MTU misconfiguration.

1. Identify the physical switch ports and VLAN interfaces backing the NSX overlay network for the affected ESX transport nodes.
2. Modify the physical network configuration to increase the MTU size to support Geneve encapsulation overhead (1700 bytes minimum). An MTU of 9000 bytes (Jumbo Frames) is highly recommended and aligns with standard VMware Cloud Foundation architecture.
3. Validate end-to-end connectivty between the Supervisor VM ESX host TEPs and Edge TEPs with a packet size of 1700 bytes.

Additional Information

MTU Guidance for NSX Transport Nodes
Install the Local Consumption Interface (LCI)