Troubleshooting network and TCP/UDP port connectivity issues on ESXi

search cancel

Troubleshooting network and TCP/UDP port connectivity issues on ESXi

book

Article ID: 341078

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information on troubleshooting network and TCP/UDP port connectivity issues using these troubleshooting tools:

ping/vmkping to troubleshoot network connectivity between two servers.
netcat (nc) to troubleshoot TCP port connectivity.
openssl to troubleshoot SSL port connectivity and verify SSL certificate information.
tcpdump-uw to collect packet traces to troubleshoot network issues.
esxcli network CLI options to view active TCP/UDP connections to the host.

Note: The nc tool helps you to check if a TCP port is online or if there may be a firewall blocking access to a TCP port.

Environment

VMware vSphere ESXi 6.0.x

VMware vSphere ESXi 6.5.x

VMware vSphere ESXi 6.7.x

VMware vSphere ESXi 7.0.x

VMware vSphere ESXi 8.0.x

Resolution

Confirming network connectivity with ping and vmkping
Confirming connectivity to a TCP port with netcat
Testing SSL port connectivity and certificate information with openssl
Collecting packet traces using tcpdump and tcpdump-uw
Viewing active TCP/UDP connections with esxcli network

Confirming network connectivity with ping and vmkping

To check if a remote host is online, you can use the ping and vmkping commands on ESXi host. The syntax of these commands are:

# ping destination-ip

# vmkping -I vmkX destination-ip

You see an output similar to:

# vmkping -I vmk2 192.xx.xx.xxx

PING 192.xx.xx.xxx (192.xx.xx.xxx): 56 data bytes

64 bytes from 192.xx.xx.xxx: icmp_seq=0 ttl=64 time=0.978 ms

64 bytes from 192.xx.xx.xxx: icmp_seq=1 ttl=64 time=1.009 ms

In this sample output, you can see that the ESXi host is able to communicate with the remote host with IP address 192.xx.xx.xxx using vmk2.

Note: On ESXi, the ping and vmkping are the same command and run from the vmkernel network stack because there is no Service Console in ESXi.

For more information on using the ping command, see Testing network connectivity with the ping command (315423).

Confirming connectivity to a TCP port with netcat

The telnet command is not available in any versions of ESXi and, therefore, you must use netcat (nc) to confirm connectivity to a TCP port on a remote host. The syntax of the nc command is:

# nc -z <destination-ip> <destination-port>

When testing connectivity to TCP port 80, you will see an output similar to:

# nc -z 192.xx.xx.xxx 80
Connection to 192.xx.xx.xxx 80 port [tcp/http] succeeded!

In the sample output, you can see that you are able to establish a connection to TCP port 80 on the host 192.xx.xx.xxx.

Note: Netcat includes an option to test UDP connectivity with the -u flag, but because UDP is a connectionless protocol, it will report as 'succeeded' even when ports are closed or blocked, and only fail when the ESXi firewall is explicitly blocking the port or cannot resolve the destination. Instead, test UDP connectivity using tcpdump (on the vCenter appliance) or tcpdump-uw (in ESXi).

The nc command can also be used to check the connectivity to a range of TCP ports on a remote host:

# nc -w 1 -z 192.xx.xx.xxx 20-81

Connection to 192.xx.xx.xxx 22 port [tcp/ssh] succeeded!

Connection to 192.xx.xx.xxx 80 port [tcp/http] succeeded!

For UDP connection check:

# nc -zu 192.xx.xx.xxx 8182

Connection to 192.xx.xx.xxx 8182 port [udp/*] succeeded!

Note: The -w option specifies a timeout value.

Note: Port scanning is a very powerful troubleshooting tool, but may be against your company network or security policies. Check with your network or security team to ensure that they are aware of this activity.

Testing SSL port connectivity and certificate information with openssl

To test SSL ports, you can use the openssl command to test connectivity and also to confirm the current SSL information. This can be useful when confirming SSL certificates with vCenter Server. The syntax of the openssl command is:

# openssl s_client -connect destination-ip:ssl-port

You see an output similar to:

# openssl s_client -connect 192.xx.xx.xxx:443

CONNECTED(00000003)

Where 443 is the default SSL port.

In this sample output, you can see that connection to the remote server 192.xx.xx.xxx over the SSL port was successful.

Note: The output may contain considerable information regarding the SSL certificates, which may be useful in troubleshooting certificate issues.

Collecting packet traces using tcpdump and tcpdump-uw

ESXi hosts come with the packet tracing tools, tcpdump-uw, which can be used to collect network traces. The network traces are useful in troubleshooting network issues.

tcpdump-uw can be used on ESXi hosts to capture packet traces from a vmkernel (vmk) interface.

For more information on using tcpdump-uw, see Capturing a network trace in ESXi using Tech Support Mode or ESXi Shell (311229).

Viewing active TCP/UDP connections with esxcli network

When troubleshooting network connectivity issues, it may be helpful to see all the active incoming and outgoing TCP/UDP connections on an ESXi host. ESXi hosts can use esxcli network to show the list of TCP/UDP connections. The commands are:

ESXi: # esxcli network ip connection list

Sample output from an ESXi host:

# esxcli network ip connection list
Proto Recv Q Send Q Local Address Foreign Address State World ID CC Algo World Name
----- ------ ------ ---------------- ------------------- ----------- -------- ------- ----------
tcp 0 0 127.0.0.1:8307 127.0.0.1:61848 ESTABLISHED 263226 newreno hostd-IO
tcp 0 1123 127.0.0.1:61848 127.0.0.1:8307 ESTABLISHED 263578 newreno envoy
tcp 0 0 127.0.0.1:80 127.0.0.1:10951 ESTABLISHED 263578 newreno envoy
tcp 0 0 127.0.0.1:10951 127.0.0.1:80 ESTABLISHED 264083 newreno python
tcp 0 0 192.XXX.XX.XX:443 192.XXX.XX.XX:42506 ESTABLISHED 263577 newreno envoy
tcp 0 0 127.0.0.1:8307 127.0.0.1:14104 ESTABLISHED 263225 newreno hostd-IO
tcp 0 0 127.0.0.1:14104 127.0.0.1:8307 ESTABLISHED 263577 newreno envoy
tcp 0 0 127.0.0.1:80 127.0.0.1:53692 ESTABLISHED 263577 newreno envoy
tcp 0 0 127.0.0.1:53692 127.0.0.1:80 ESTABLISHED 263973 newreno worker
tcp 0 0 127.0.0.1:8307 127.0.0.1:26502 ESTABLISHED 263225 newreno hostd-IO
tcp 0 0 127.0.0.1:26502 127.0.0.1:8307 ESTABLISHED 263577 newreno envoy
tcp 0 0 192.XXX.XX.XX:22 192.XXX.XX.XX:42470 ESTABLISHED 262661 newreno busybox
tcp 0 0 192.XXX.XX.XX:22 192.XXX.XX.XX:42466 ESTABLISHED 262661 newreno busybox
tcp 0 0 127.0.0.1:8307 127.0.0.1:49365 ESTABLISHED 263226 newreno hostd-IO
tcp 0 0 127.0.0.1:49365 127.0.0.1:8307 ESTABLISHED 263576 newreno envoy
tcp 0 0 127.0.0.1:8307 127.0.0.1:49364 ESTABLISHED 263226 newreno hostd-IO
tcp 0 0 127.0.0.1:49364 127.0.0.1:8307 ESTABLISHED 263576 newreno envoy
tcp 0 0 127.0.0.1:8307 127.0.0.1:49363 ESTABLISHED 263226 newreno hostd-IO
tcp 0 0 127.0.0.1:49363 127.0.0.1:8307 ESTABLISHED 263576 newreno envoy
tcp 0 0 127.0.0.1:8307 127.0.0.1:49360 ESTABLISHED 263226 newreno hostd-IO
tcp 0 0 127.0.0.1:49360 127.0.0.1:8307 ESTABLISHED 263576 newreno envoy
tcp 0 0 127.0.0.1:8307 127.0.0.1:49359 ESTABLISHED 263225 newreno hostd-IO
tcp 0 0 127.0.0.1:49359 127.0.0.1:8307 ESTABLISHED 263576 newreno envoy
tcp 0 0 127.0.0.1:12000 0.0.0.0:0 LISTEN 263322 newreno vpxa-worker
tcp 0 0 127.0.0.1:8307 127.0.0.1:59381 ESTABLISHED 263226 newreno hostd-IO
tcp 0 719 127.0.0.1:59381 127.0.0.1:8307 ESTABLISHED 263754 newreno vpxa-worker
tcp 0 0 127.0.0.1:8307 127.0.0.1:27943 ESTABLISHED 263226 newreno hostd-IO
tcp 0 0 127.0.0.1:27943 127.0.0.1:8307 ESTABLISHED 263336 newreno vpxa-worker
tcp 0 1106 127.0.0.1:8307 127.0.0.1:41063 ESTABLISHED 263226 newreno hostd-IO
tcp 0 0 127.0.0.1:41063 127.0.0.1:8307 ESTABLISHED 263326 newreno vpxa-worker
tcp 0 0 127.0.0.1:443 127.0.0.1:20745 ESTABLISHED 263576 newreno envoy
tcp 0 0 127.0.0.1:20745 127.0.0.1:443 ESTABLISHED 263336 newreno vpxa-worker
tcp 0 0 127.0.0.1:8307 127.0.0.1:14165 ESTABLISHED 263226 newreno hostd-IO
tcp 0 0 127.0.0.1:14165 127.0.0.1:8307 ESTABLISHED 263576 newreno envoy
tcp 0 0 [::]:80 [::]:0 LISTEN 262897 newreno envoy
tcp 0 0 0.0.0.0:80 0.0.0.0:0 LISTEN 262897 newreno envoy
tcp 0 0 [::]:443 [::]:0 LISTEN 262897 newreno envoy
tcp 0 0 0.0.0.0:443 0.0.0.0:0 LISTEN 262897 newreno envoy
tcp 0 0 [::1]:9131 [::]:0 LISTEN 263164 newreno hostd
tcp 0 0 127.0.0.1:9131 0.0.0.0:0 LISTEN 263164 newreno hostd
tcp 0 0 [::1]:12001 [::]:0 LISTEN 263164 newreno hostd
tcp 0 0 127.0.0.1:12001 0.0.0.0:0 LISTEN 263164 newreno hostd
tcp 0 0 [::1]:9127 [::]:0 LISTEN 263164 newreno hostd
tcp 0 0 127.0.0.1:9127 0.0.0.0:0 LISTEN 263164 newreno hostd
tcp 0 0 [::1]:9129 [::]:0 LISTEN 263164 newreno hostd
tcp 0 0 127.0.0.1:9129 0.0.0.0:0 LISTEN 263164 newreno hostd
tcp 0 0 [::1]:8307 [::]:0 LISTEN 263164 newreno hostd
tcp 0 0 127.0.0.1:8307 0.0.0.0:0 LISTEN 263164 newreno hostd
tcp 0 0 [::1]:8309 [::]:0 LISTEN 263164 newreno hostd
tcp 0 0 127.0.0.1:8309 0.0.0.0:0 LISTEN 263164 newreno hostd
tcp 0 0 [::1]:8083 [::]:0 LISTEN 263284 newreno settingsd
tcp 0 0 127.0.0.1:8083 0.0.0.0:0 LISTEN 263284 newreno settingsd
tcp 0 0 127.0.0.1:8089 0.0.0.0:0 LISTEN 263289 newreno vpxa
tcp 0 0 [::1]:9984 [::]:0 LISTEN 263296 newreno healthd
tcp 0 0 127.0.0.1:9984 0.0.0.0:0 LISTEN 263296 newreno healthd
tcp 0 0 [::1]:8098 [::]:0 LISTEN 263128 newreno apiForwarder
tcp 0 0 127.0.0.1:8098 0.0.0.0:0 LISTEN 263128 newreno apiForwarder
tcp 0 0 [::1]:9199 [::]:0 LISTEN 263008 newreno esxtokend
tcp 0 0 127.0.0.1:9199 0.0.0.0:0 LISTEN 263008 newreno esxtokend
tcp 0 0 [::]:902 [::]:0 LISTEN 263074 newreno nfcd
tcp 0 0 0.0.0.0:902 0.0.0.0:0 LISTEN 263074 newreno nfcd
tcp 0 0 [::1]:549 [::]:0 LISTEN 263010 newreno rhttpproxy
tcp 0 0 127.0.0.1:549 0.0.0.0:0 LISTEN 263010 newreno rhttpproxy
tcp 0 0 [::1]:8303 [::]:0 LISTEN 262994 newreno hostdCgiServer
tcp 0 0 127.0.0.1:8303 0.0.0.0:0 LISTEN 262994 newreno hostdCgiServer
tcp 0 0 [::]:9080 [::]:0 LISTEN 262953 newreno ioFilterVPServer
tcp 0 0 [::1]:8888 [::]:0 LISTEN 262878 newreno vdtc
tcp 0 0 127.0.0.1:8888 0.0.0.0:0 LISTEN 262878 newreno vdtc
tcp 0 0 [::1]:9999 [::]:0 LISTEN 262839 newreno esxgdpd
tcp 0 0 127.0.0.1:9999 0.0.0.0:0 LISTEN 262839 newreno esxgdpd
tcp 0 0 [::]:22 [::]:0 LISTEN 262661 newreno busybox
tcp 0 0 0.0.0.0:22 0.0.0.0:0 LISTEN 262661 newreno busybox
tcp 0 0 [::]:8000 [::]:0 LISTEN 262368 newreno
tcp 0 0 [::1]:7890 [::]:0 LISTEN 262645 newreno kmxa
tcp 0 0 127.0.0.1:7890 0.0.0.0:0 LISTEN 262645 newreno kmxa
tcp 0 0 [::]:8300 [::]:0 LISTEN 262368 newreno
udp 0 0 0.0.0.0:0 0.0.0.0:0 263289 vpxa
udp 0 0 0.0.0.0:0 0.0.0.0:0 263164 hostd
udp 0 0 [::1]:6831 [::]:0 262878 vdtc
udp 0 0 127.0.0.1:6831 0.0.0.0:0 262878 vdtc

To retrieve errors and statistics for a network adapter, run this command:

# esxcli network nic stats get -n <vmnicX>

Where <vmnicX> is the name of a NIC in your ESXi host.

Note: These counters are tracked by the physical NIC driver that the physical NIC driver develops, maintains and provides support for. Therefore if you see any packet drops reported or error counters in the above output, make a support request with your server vendor for further investigation and configuration recommendations.

For more information on these counters, see Troubleshooting network receive traffic faults and other NIC errors in ESXi (341594).

Additional Information

For more information about gathering information and troubleshooting network issues with esxcli network commands, see the Network Troubleshooting section in the vSphere Command-Line Interface Concepts and Examples guide.
For more troubleshooting articles, see:

Feedback

thumb_up Yes

thumb_down No