Troubleshooting network and TCP/UDP port connectivity issues on ESXi

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information on troubleshooting network and TCP/UDP port connectivity issues using these troubleshooting tools:

ping/vmkping to troubleshoot network connectivity between two servers.
netcat (nc) to troubleshoot TCP port connectivity.
openssl to troubleshoot SSL port connectivity and verify SSL certificate information.
tcpdump-uw to collect packet traces to troubleshoot network issues.
esxcli network CLI options to view active TCP/UDP connections to the host.

Note: The nc tool helps you to check if a TCP port is online or if there may be a firewall blocking access to a TCP port.

Resolution

The telnet command is not available in any versions of ESXi and, therefore, you must use netcat (nc) to confirm connectivity to a TCP port on a remote host. The syntax of the nc command is:

# nc -z <destination-ip> <destination-port>

When testing connectivity to TCP port 80, you will see an output similar to:

# nc -z 192.xx.xx.xxx 80
Connection to 192.xx.xx.xxx 80 port [tcp/http] succeeded!

In the sample output, you can see that you are able to establish a connection to TCP port 80 on the host 192.xx.xx.xxx.

Note: Netcat includes an option to test UDP connectivity with the -u flag, but because UDP is a connectionless protocol, it will report as 'succeeded' even when ports are closed or blocked, and only fail when the ESXi firewall is explicitly blocking the port or cannot resolve the destination. Instead, test UDP connectivity using tcpdump (on the vCenter appliance) or tcpdump-uw (in ESXi).

The nc command can also be used to check the connectivity to a range of TCP ports on a remote host:

# nc -w 1 -z 192.xx.xx.xxx 20-81

Connection to 192.xx.xx.xxx 22 port [tcp/ssh] succeeded!

Connection to 192.xx.xx.xxx 80 port [tcp/http] succeeded!

Note: The -w option specifies a timeout value.

Note: Port scanning is a very powerful troubleshooting tool, but may be against your company network or security policies. Check with your network or security team to ensure that they are aware of this activity.

Testing SSL port connectivity and certificate information with openssl

To test SSL ports, you can use the openssl command to test connectivity and also to confirm the current SSL information. This can be useful when confirming SSL certificates with vCenter Server. The syntax of the openssl command is:

# openssl s_client -connect destination-ip:ssl-port

You see an output similar to:

# openssl s_client -connect 192.xx.xx.xxx:443

CONNECTED(00000003)

Where 443 is the default SSL port.

In this sample output, you can see that connection to the remote server 192.xx.xx.xxx over the SSL port was successful.

Note: The output may contain considerable information regarding the SSL certificates, which may be useful in troubleshooting certificate issues.

Collecting packet traces using tcpdump and tcpdump-uw

ESXi hosts come with the packet tracing tools, tcpdump-uw, which can be used to collect network traces. The network traces are useful in troubleshooting network issues.

tcpdump-uw can be used on ESXi hosts to capture packet traces from a vmkernel (vmk) interface.

For more information on using tcpdump-uw, see Capturing a network trace in ESXi using Tech Support Mode or ESXi Shell (311229).

Viewing active TCP/UDP connections with esxcli network

When troubleshooting network connectivity issues, it may be helpful to see all the active incoming and outgoing TCP/UDP connections on an ESXi host. ESXi hosts can use esxcli network to show the list of TCP/UDP connections. The commands are:

ESXi: # esxcli network ip connection list

Sample output from an ESXi host:

# esxcli network connection list

Proto Recv-Q Send-Q Local Address Foreign Address State World ID

tcp 0 52 192.xx.xx.xxx:22 192.xx.xx.xxx:55169 ESTABLISHED 0

tcp 0 0 127.0.0.1:62024 127.0.0.1:5988 TIME_WAIT 0

tcp 0 0 127.0.0.1:57867 127.0.0.1:5988 TIME_WAIT 0

tcp 0 0 127.0.0.1:62196 127.0.0.1:5988 TIME_WAIT 0

tcp 0 0 127.0.0.1:8307 127.0.0.1:52943 ESTABLISHED 5790

tcp 0 0 127.0.0.1:52943 127.0.0.1:8307 ESTABLISHED 5790

tcp 0 0 127.0.0.1:80 127.0.0.1:55629 ESTABLISHED 5785

tcp 0 0 127.0.0.1:55629 127.0.0.1:80 ESTABLISHED 6613

tcp 0 0 127.0.0.1:8307 127.0.0.1:56319 ESTABLISHED 5785

tcp 0 0 127.0.0.1:56319 127.0.0.1:8307 ESTABLISHED 5785

tcp 0 0 127.0.0.1:80 127.0.0.1:62782 ESTABLISHED 5166

tcp 0 0 127.0.0.1:62782 127.0.0.1:80 ESTABLISHED 6613

tcp 0 0 127.0.0.1:5988 127.0.0.1:53808 FIN_WAIT_2 0

tcp 0 0 127.0.0.1:53808 127.0.0.1:5988 CLOSE_WAIT 5166

tcp 0 0 127.0.0.1:8307 127.0.0.1:56963 CLOSE_WAIT 5788

tcp 0 0 127.0.0.1:56963 127.0.0.1:8307 FIN_WAIT_2 5785

tcp 0 0 127.0.0.1:8307 0.0.0.0:0 LISTEN 5031

tcp 0 0 127.0.0.1:8309 0.0.0.0:0 LISTEN 5031

tcp 0 0 127.0.0.1:5988 0.0.0.0:0 LISTEN 0

tcp 0 0 0.0.0.0:5989 0.0.0.0:0 LISTEN 0

tcp 0 0 0.0.0.0:80 0.0.0.0:0 LISTEN 5031

tcp 0 0 0.0.0.0:443 0.0.0.0:0 LISTEN 5031

tcp 0 0 127.0.0.1:12001 0.0.0.0:0 LISTEN 5031

tcp 0 0 127.0.0.1:8889 0.0.0.0:0 LISTEN 5331

tcp 0 0 192.xx.xx.xxx:427 0.0.0.0:0 LISTEN 0

tcp 0 0 127.0.0.1:427 0.0.0.0:0 LISTEN 0

tcp 0 0 0.0.0.0:22 0.0.0.0:0 LISTEN 0

tcp 0 0 0.0.0.0:902 0.0.0.0:0 LISTEN 0

tcp 0 0 0.0.0.0:8000 0.0.0.0:0 LISTEN 4801

tcp 0 0 0.0.0.0:8100 0.0.0.0:0 LISTEN 4795

udp 0 0 192.xx.xx.xxx:427 0.0.0.0:0 0

udp 0 0 0.0.0.0:427 0.0.0.0:0 0

udp 0 0 192.xx.xx.xxx:68 0.0.0.0:0 4693

udp 0 0 0.0.0.0:8200 0.0.0.0:0 4795

udp 0 0 0.0.0.0:8301 0.0.0.0:0 4686

udp 0 0 0.0.0.0:8302 0.0.0.0:0 4686

To retrieve errors and statistics for a network adapter, run this command:

# esxcli network nic stats get -n <vmnicX>

Where <vmnicX> is the name of a NIC in your ESXi host.

Note: These counters are tracked by the physical NIC driver that the physical NIC driver develops, maintains and provides support for. Therefore if you see any packet drops reported or error counters in the above output, make a support request with your server vendor for further investigation and configuration recommendations.

For more information on these counters, see Troubleshooting network receive traffic faults and other NIC errors in ESXi (341594).