vCenter upgrade fails with an error "Error: IP already exists in the network"

Products

• VMware vCenter Server

Issue/Introduction

• vCenter upgrade fails at Stage 2 of the process with an error message "Error: IP already exists in the network. Unable to set the network parameters"
• In the file, var/log/vmware/upgrade/upgrade-export.log you see entries similar to:

YYYY-MM-DDTHH:MIN INFO upgrade_commands Shutting down source machine [vCenter FQDN]
YYYY-MM-DDTHH:MIN INFO networking_utils isHostReachable(): getaddrinfo() found 1 entries (first is used): FAMILY: AddressFamily.AF_INET, TYPE=SocketKind.SOCK_STREAM, PROTO=6, CANONNAME=, ADDR=('vCenter IP', 22)
YYYY-MM-DDTHH:MIN ERROR networking_utils isHostReachable() failed: [TimeoutError]: "timed out", treating as unreachable host
YYYY-MM-DDTHH:MIN INFO upgrade_commands Source machine is down, waiting for 60 additional seconds to make sure it is completely down.
YYYY-MM-DDTHH:MIN INFO upgrade_commands Shutdown completed successfully.

• In the file, /var/log/vmware/upgrade/export.json you see entries similar to:

YYYY-MM-DDTHH:MIN  Stderr: YYYY-MM-DD HH:MIN [ERROR] Cannot run /sbin/ifup eth0 command. Unknown error. Return code:256 output: Make sure the interface is down or not assigned any IP
eth0 is DOWN or not assigned an IP. Bringnig eth0 up...
Can not find the manual filename, let us search for the auto filename
Performing duplicate address check for IPv4 address <vCenter IP address>
Error: IP already exists in the network
Unable to set the network parameters

Environment

VMware vCenter Server

Cause

• When the source vCenter is still shutting down (holding the source IP address) and the target vCenter is in the process of updating/switching the same IP address from Temporary IP address at the same Time.
• As the error mentions, there is a probability of duplicate IP address and hence the upgrade process fails at Stage 2. Network devices such as switch or router, hold duplicate IP associated to MAC address or vice versa.

Resolution

Follow the below steps if the issue occurs when the upgrade fails while the source vCenter is still shutting down:

• Take Snapshot of the source vCenter (Offline Snapshots of all the vCenters in case of linked mode).
• Take SSH of the source vCenter and run the below commands:
  
  
  mv /usr/sbin/shutdown /usr/sbin/shutdown-backup
  
  echo -e '#!/bin/bash\nsystemctl start systemd-networkd\nip link set eth0 down\nip link set eth1 down\n/usr/sbin/shutdown-backup -h now' > /usr/sbin/shutdown
  
  chmod +x /usr/sbin/shutdown
  
  
• Restart the upgrade process.

Note: If the above does not work, there may be another device with the same source vCenter IP on the network.

• Steps to determine duplicate IP/MAC address:
  • Shutdown the vCenter server.
  • SSH to any Linux virtual machine
    • arping -D -I eth0 -c 2 <vCenter IP address>
    • echo $?
• Example Output for arping command:

ARPING <vCenter IP address> from <Linux machine IP> ens192
Unicast reply from <vCenter IP address> [MAC Address 1] 0.751ms
Unicast reply from <vCenter IP address> [MAC Address 2] 1.768ms

• Output of echo $ is 1 when duplicate MAC address is detected.

To explain, the above output shows two MAC address mentioned for one vCenter IP. Correlate the MAC address in use by vCenter server from ifconfig command and contact on-premises network administrator. Once done, reinitiate vCenter upgrade.

Additional Information

This can also occur if there is a network issue between the new and old vCenters. Retry the upgrade, but make sure to place the new vCenter on the same host as the old vCenter in order to assure that the transfer operations do not use the physical network and the connection does not get "timed out".