Host cannot download the VIBs from vCenter using custom port

Products

• VMware NSX

Issue/Introduction

• In EAM logs, similar entries to below can be seen:
  
  2017-07-31T11:00:14Z esxupdate: esxupdate: ERROR: MetadataDownloadError: ('http://#.#.#.#:888/eam/vib?id=2d526b23-81c1-4960-####-#############-0', None, "('http://#.#.#.#:888/eam/vib?id=2d526b23-81c1-####-#############-0', '/tmp/tmpQGxL0w', '[Errno 12] Timeout: <urlopen error timed out>')")
   
• On the ESXi host, query the URL manually to confirm the timeout:
  
  [root@HQ-ESXi-Prod-01a:~] cd /tmp [root@HQ-ESXi-Prod-01a:/tmp] wget http://#.#.#.#:888/eam/vib?id=2d526b23-81c1-####-#############-0 Connecting to #.#.#.#:888 (#.#.#.#:888) wget: can't connect to remote host (#.#.#.#): Connection timed out
  
  Note: If a HTTP 404, 301 or anything like that is received, it's a response sent by the vCenter Server, not a timeout.
   
• Use nc to test the connection between the ESXi host and vCenter Server.
  
  [root@HQ-ESXi-Prod-01a:~] nc -vz #.#.#.# 888 nc: connect to #.#.#.# port 888 (tcp) failed: Connection timed out
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.3.x

Resolution

To resolve this issue, disable the firewall temporarily to allow the host to download the VIB's.

1. Disable the firewall:
   
   esxcli network firewall get
esxcli network firewall set --enabled false
esxcli network firewall unload
esxcli network firewall get
    
2. Resolve the alerts on Guest Introspection / Agency.
    
3. Enable the firewall after the VIB's download.
   
   esxcli software vib list | grep mux
esxcli network firewall load
esxcli network firewall set --enabled true
esxcli network firewall get