vCenter upgrade fails with an error "Error: IP already exists in the network" though the source vCenter server shutdown successfully

Products

• VMware vCenter Server

Issue/Introduction

vCenter upgrade fails at Stage 2 of the process with an error message "Error: IP already exists in the network. Unable to set the network parameters"

• In the file, var/log/vmware/upgrade/upgrade-export.log you see entries similar to:

YYYY-MM-DDTHH:MIN INFO upgrade_commands Shutting down source machine [vCenter FQDN]
YYYY-MM-DDTHH:MIN INFO networking_utils isHostReachable(): getaddrinfo() found 1 entries (first is used): FAMILY: AddressFamily.AF_INET, TYPE=SocketKind.SOCK_STREAM, PROTO=6, CANONNAME=, ADDR=('vCenter IP', 22)
YYYY-MM-DDTHH:MIN ERROR networking_utils isHostReachable() failed: [TimeoutError]: "timed out", treating as unreachable host
YYYY-MM-DDTHH:MIN INFO upgrade_commands Source machine is down, waiting for 60 additional seconds to make sure it is completely down.
YYYY-MM-DDTHH:MIN INFO upgrade_commands Shutdown completed successfully.

• In the file, /var/log/vmware/upgrade/export.json you see entries similar to:

YYYY-MM-DDTHH:MIN  Stderr: YYYY-MM-DD HH:MIN [ERROR] Cannot run /sbin/ifup eth0 command. Unknown error. Return code:256 output: Make sure the interface is down or not assigned any IP
eth0 is DOWN or not assigned an IP. Bringnig eth0 up...
Can not find the manual filename, let us search for the auto filename
Performing duplicate address check for IPv4 address <vCenter IP address>
Error: IP already exists in the network
Unable to set the network parameters

Cause

As the error mentions, there is a probability of duplicate IP address and hence the upgrade process fails at Stage 2. Network devices such as switch or router, hold duplicate IP associated to MAC address or vice versa.

Resolution

Below mentioned step requires downtime for vCenter server. Proceed with planned maintenance in place.

• Steps to determine duplicate IP/MAC address:
  • Shutdown the vCenter server.
  • SSH to any Linux virtual machine
    • arping -D -I eth0 -c 2 <vCenter IP address>
    • echo $?
• Example Output for arping command:

ARPING <vCenter IP address> from <Linux machine IP> ens192
Unicast reply from <vCenter IP address> [MAC Address 1] 0.751ms
Unicast reply from <vCenter IP address> [MAC Address 2] 1.768ms

• Output of echo $ is 1 when duplicate MAC address is detected.

To explain, the above output shows two MAC address mentioned for one vCenter IP. Correlate the MAC address in use by vCenter server from ifconfig command and contact on-premises network administrator. Once done, reinitiate vCenter upgrade.