vCenter Upgrade fails during Install Phase from SDDC Manager with UI Error: "Failed to Refresh Public API Access Token using refresh token: Encountered error trying to refresh public api access token using refresh token: Identify internal Server Error"

Products

• VMware SDDC Manager
• VMware vCenter Server

Issue/Introduction

• vCenter Server upgrade initiated via SDDC Manager fails during the "VCENTER UPGRADE INSTALL" phase.
• The following error is observed on the SDDC Manager UI:

"Failed to refresh public api access token using refresh token: Encountered error trying to refresh public api access token using refresh token: Identify internal Server Error"

•  While checking the target vCenter server appliance console, it shows ERROR:  Upgrade EXPORT failed.

• vCenter upgrade fails at Stage 2 of the process with an error message "Error: IP already exists in the network. Unable to set the network parameters".

• In the /var/log/vmware/upgrade/upgrade-export.log file, you see entries similar to:
  apply_networking.UserUpgradeNetworkConfigError: [Error setting network. Details : DD/MM/YYYY [ERROR] Cannot run /sbin/ifup eth0 command. Unknown error. Return code : 256 output: Make sure the interface is down or not assigned any IP
eth0 is DOWN or not assigned an IP. Bringing eth0 up...
Can not find the manual filename, let us search for the auto filename
Performing duplicate address check for IPv4 address XX.XX.XX.XX
IP already exists in the network
Unable to set the network parameters
Failed to set network]
YYYY-MM-DD ERROR __main__ ERROR: Fatal error during upgrade EXPORT. For more details take a look at: /var/log/vmware/upgrade/export-upgrade-runner.log
  
  
• In the /var/log/vmware/upgrade/export.json file, you see entries similar to:
  YYYY-MM-DD [ERROR] Cannot run /sbin/ifup eth0 command. Unknown error. Return code : 256 output: Make sure the interface is down or not assigned any IP\neth0 is DOWN or not assigned an IP. Bringnig eth0 up...\nCan not find the manual filename, let us search for the auto filename\nPerforming duplicate address check for IPv4 address XX.XX.XX.XX\n\u00##[1;31mError: IP already exists in the network\u00##[0;39m\n\nUnable to set the network parameters\n"
],
"localized": "Error setting network. Details : YYYY-MM-DD [ERROR] Cannot run /sbin/ifup eth0 command. Unknown error. Return code : 256 output: Make sure the interface is down or not assigned any IP\neth0 is DOWN or not assigned an IP. Bringing eth0 up...\nCan not find the manual filename, let us search for the auto filename\nPerforming duplicate address check for IPv4 address XX.XX.XX.XX\n\u00##[1;31mError: IP already exists in the network\u00##[0;39m\n\nUnable to set the network parameters\n"

Environment

• VMware Cloud Foundation
• VMware vCenter Server

Cause

This issue can occur due to any of the following reasons:

• When the source vCenter is still shutting down (holding the source IP address) and the target vCenter is in the process of updating/switching the same IP address from Temporary IP address at the same Time.
• The presence of duplicate IP in the network where the IP address of the source vCenter is assigned to another device in the network

Resolution

Follow the below steps if the issue occurs when the upgrade fails while the source vCenter is still shutting down:

• Take snapshot of the SDDC manager.
• Take vCenter server snapshot. (Offline Snapshots of all the vCenters in case of linked mode).
• Take SSH of the source vCenter and run the below commands:

# mv /usr/sbin/shutdown /usr/sbin/shutdown-backup
# echo -e '#!/bin/bash\nsystemctl start systemd-networkd\nip link set eth0 down\nip link set eth1 down\n/usr/sbin/shutdown-backup -h now' > /usr/sbin/shutdown
# chmod +x /usr/sbin/shutdown

• Restart the vCenter upgrade process from SDDC manager.

Note: If the above does not work, there may be another device with the same source vCenter IP on the network.

• Steps to determine duplicate IP/MAC address:

         # arping -D -I eth0 -c 2 <vCenter IP address> # echo $? 

• Shutdown the vCenter server.
• SSH to any Linux virtual machine
• Example Output for arping command:

   # ARPING <vCenter IP address> from <Linux machine IP> ens192

    Unicast reply from <vCenter IP address> [MAC Address 1] 0.751ms

    Unicast reply from <vCenter IP address> [MAC Address 2] 1.768ms

• Output of echo $ is 1 when duplicate MAC address is detected.

To explain, the above output shows two MAC address mentioned for one vCenter IP. Correlate the MAC address in use by vCenter server from ifconfig command and contact on-premises network administrator. Once done, reinitiate vCenter upgrade.

Additional Information

vCenter upgrade fails with an error "Error: IP already exists in the network"