Troubleshooting Tools for Networking on vCenter Server Appliance 6.5 and Above

Products

VMware vCenter Server

Issue/Introduction

vCenter Server Appliance 6.5 and above is built using VMware Photon OS. Previous versions of the vCenter Server Appliance were built using Suse Linux Enterprise Server (SLES).

This article explains some of the basic commands that may be useful when troubleshooting network related issues for Photon OS for vCenter Server Appliance 6.5 and above.

Important: This article is explicitly for vCenter Server Appliance 6.5 and above and may not work for previous versions.

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x

Resolution

Use of commands IP and SS instead of IPconfig and netstat.

Although the ifconfig command and the netstat command work on Photon OS, VMware recommends using the ip or ss commands. The ifconfig and netstat commands are deprecated.

Photon OS Command	SLES Command
`ip addr`	ifconfig -a
ss	netstat
ip route	route
ip maddr	netstat -g
ip link set eth0 up	ifconfig eth0 up
ip -s neigh	arp -v
ip link set eth0 mtu 9000	ifconfig eth0 mtu 9000

Verifying the Status of Network up-link using the command:

The command networkctl shows information about network connections that helps you configure networking services and troubleshoot networking problems.

For example:
```
# networkctl
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
```
Two up-links listed.

The command networkctl status displays information about the network up-link status

For example:

# networkctl status
State: routable
Address: 10.####.####.#### on eth0
Gateway: 10.####.####.#### (Cisco Systems, Inc) on eth0
DNS: 10.####.####.####
10.####.####.####
Domain: example.com

To display the up-link configuration, use the command networkctl status etho

For example:

# networkctl status eth0

2: eth0
Link File: /usr/lib/systemd/network/99-default.link
Network File: /etc/systemd/network/10-eth0.network
Type: ether
State: routable (configured)
Path: pci-0000:03:00.0
Driver: vmxnet3
Vendor: VMware
Model: VMXNET3 Ethernet Controller
HW Address: 00:00:00:00:00:00
MTU: 1500
Address:10.####.####.####
Gateway: 10.####.####.#### (Cisco Systems, Inc)
DNS: 10.####.####.####
10.####.####.####
Domain: example.com
CLIENTID: 00:00:00:00:00:00

Using the command TCPDUMP:

The tcpdump Command captures and analyzes packets on the network interface. Tcpdump is available by default on the vCenter Server Appliance 6.5

When running the #tcpdump --help command, review the output:

#tcpdump --help 
tcpdump version 4.7.4
libpcap version 1.7.4
Usage: tcpdump [-aAbdDefhHIJKlLnNOpqRStuUvxX#] [ -B size ] [ -c count ]
[ -C file_size ] [ -E algo:secret ] [ -F file ] [ -G seconds ]
[ -i interface ] [ -j tstamptype ] [ -M secret ] [ --number ]
[ -Q in|out|inout ]
[ -r file ] [ -s snaplen ] [ --time-stamp-precision precision ]
[ --immediate-mode ] [ -T type ] [ --version ] [ -V file ]
[ -w file ] [ -W filecount ] [ -y datalinktype ] [ -z command ]
[ -Z user ] [ expression ]

Using the command NETCAT:

The netcat command can be used to test port connectivity.

When running the #netcat -h command, review the output similar to:

#netcat -h
GNU netcat 0.7.1, a rewrite of the famous networking tool.
Basic usages:
connect to somewhere: netcat [options] hostname port [port] ...
listen for inbound: netcat -l -p port [options] [hostname] [port] ...
tunnel to somewhere: netcat -L hostname:port -p port [options]


Mandatory arguments to long options are mandatory for short options too.
Options:
-c, --close close connection on EOF from stdin
-e, --exec=PROGRAM program to exec after connect
-g, --gateway=LIST source-routing hop point[s], up to 8
-G, --pointer=NUM source-routing pointer: 4, 8, 12, ...
-h, --help display this help and exit
-i, --interval=SECS delay interval for lines sent, ports scanned
-l, --listen listen mode, for inbound connects
-L, --tunnel=ADDRESS:PORT forward local port to remote address
-n, --dont-resolve numeric-only IP addresses, no DNS
-o, --output=FILE output hexdump traffic to FILE (implies -x)
-p, --local-port=NUM local port number
-r, --randomize randomize local and remote ports
-s, --source=ADDRESS local source address (ip or hostname)
-t, --tcp TCP mode (default)
-T, --telnet answer using TELNET negotiation
-u, --udp UDP mode
-v, --verbose verbose (use twice to be more verbose)
-V, --version output version information and exit
-x, --hexdump hexdump incoming and outgoing traffic
-w, --wait=SECS timeout for connects and final net reads
-z, --zero zero-I/O mode (used for scanning)
Remote port number can also be specified as range. Example: '1-1024'

Using the command LSOF:

The lsof command and can be used to view a list of running processes and the ports they are consuming.
When running to #lsof -i -P -n command,review the output similar to:

# lsof -i -P -n 
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 83u IPv6 11427 0t0 UDP *:69
ntpd 909 ntp 16u IPv6 11786 0t0 UDP *:123
ntpd 909 ntp 17u IPv4 11789 0t0 UDP *:123
ntpd 909 ntp 18u IPv4 11793 0t0 UDP 127.0.0.1:123
ntpd 909 ntp 19u IPv6 11795 0t0 UDP [::1]:123
ntpd 909 ntp 21u IPv4 26108 0t0 UDP 10.x.x.x:123
systemd-n 4146 systemd-network 13u IPv4 25130 0t0 UDP 10.x.x.x:68
rpcbind 4375 rpc 6u IPv4 27963 0t0 UDP *:111
rpcbind 4375 rpc 7u IPv4 27964 0t0 UDP *:731
rpcbind 4375 rpc 8u IPv4 27965 0t0 TCP *:111 (LISTEN)
rpcbind 4375 rpc 9u IPv6 27966 0t0 UDP *:111
rpcbind 4375 rpc 10u IPv6 27967 0t0 UDP *:731
rpcbind 4375 rpc 11u IPv6 27968 0t0 TCP *:111 (LISTEN)
dcerpcd 4609 root 3u IPv4 28201 0t0 TCP *:135 (LISTEN)
dcerpcd 4609 root 6u IPv4 28205 0t0 UDP *:135
dcerpcd 4609 root 7u IPv4 95629 0t0 TCP 10.####.####.####:39766->10.####.####.####:2020 (ESTABLISHED)
dcerpcd 4609 root 8u IPv4 95631 0t0 TCP 10.####.####.####:36094->10.####.####.####:2012 (ESTABLISHED)
dcerpcd 4609 root 9u IPv4 96413 0t0 TCP 110.####.####.####:43905->10.####.####.####:2015 (ESTABLISHED)
vmafdd 4616 root 3u IPv4 95630 0t0 TCP 110.####.####.####:2020->10.####.####.####:39766 (ESTABLISHED)
vmafdd 4616 root 21u IPv4 28229 0t0 TCP *:2020 (LISTEN)
vmcad 4862 root 19u IPv4 28592 0t0 TCP *:2014 (LISTEN)
vmdird 5193 root 22u IPv4 29202 0t0 TCP *:2012 (LISTEN)
vmdird 5193 root 25u IPv4 86892 0t0 TCP 10.####.####.####:389->10.####.####.####:42080 (ESTABLISHED)
vmdird 5193 root 26u IPv4 31970 0t0 TCP *:389 (LISTEN)
vmdird 5193 root 27u IPv4 31971 0t0 TCP *:11711 (LISTEN)
vmdird 5193 root 28u IPv4 29268 0t0 TCP *:636 (LISTEN)
vmdird 5193 root 29u IPv4 29269 0t0 TCP *:11712 (LISTEN)
vmdird 5193 root 32u IPv4 31973 0t0 TCP *:88 (LISTEN)
vmdird 5193 root 33u IPv4 31975 0t0 UDP *:88
vmdird 5193 root 34u IPv4 95632 0t0 TCP 10.####.####.####2012->10.####.####.####:36094 (ESTABLISHED)
vmdird 5193 root 35u IPv4 91988 0t0 TCP 10.####.####.####:389->10.####.####.####:42456 (ESTABLISHED)

Using the command systemd-networkd:

You can check its status of the network service by running the # systemctl status systemd-networkd command:

Review the output similar to:

# systemctl status systemd-networkd
systemd-networkd.service - Network Service
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2016-09-28 09:23:08 UTC; 9min ago
Docs: man:systemd-networkd.service(8)
Main PID: 4146 (systemd-network)
Status: "Processing requests..."
Tasks: 1
Memory: 276.0K
CPU: 10ms
CGroup: /system.slice/systemd-networkd.service
4146 /lib/systemd/systemd-networkd


09:23:08 localhost.localdom systemd[1]: Starting Network Service...
09:23:08 localhost.localdom systemd-networkd[4146]: Enumeration completed
09:23:08 localhost.localdom systemd[1]: Started Network Service.
09:23:08 localhost.localdom systemd-networkd[4146]: eth0: Gained carrier
09:23:11 localhost.localdom systemd-networkd[4146]: eth0: DHCPv4 address 10.####.####.#####/# via 10.####.####.####
09:23:11 localhost.localdom systemd-networkd[4146]: eth0: Configured

The network configuration of the eth0 interface is stored in the /etc/systemd/network/10-eth0.network file.

You can check the content by running the # cat /etc/systemd/network/10-eth0.network command

Review the output similar to:

# cat /etc/systemd/network/10-eth0.network

[Match]
Name=eth0
[Network]
Gateway=10.####.####.####
Address=10.####.####.####/#
DHCP=no
[DHCP]
UseDNS=false