Error: "Internal error occurs during execution of upgrade process" when upgrading to vCenter Server Appliance 6.x

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server upgrade fails with the error:
  
  Error
Internal error occurs during execution of upgrade process.

Resolution
Send upgrade log files to VMware technical support team for further assistance.
   
• In the /var/log/vmware/upgrade/bootstrap.log file on the destination vCenter Server Appliance, there are entries similar to:
  
  YYYY-MM-DDTHH:MM:SS.457Z INFO ssh_utils Local directory /usr/lib/vmware/cis_upgrade_runner has been successfully uploaded to root@vcenter_fqdn.local:/tmp/vmware-root/tmpvmware72
YYYY-MM-DDTHH:MM:SS.497Z INFO base_commands Directory /var/log/vmware/upgrade is successfully removed
YYYY-MM-DDTHH:MM:SS.497Z INFO base_commands Executing command: [<class 'upgrade_commands.UpgradeRequirementsLinuxCommand'>]
YYYY-MM-DDTHH:MM:SS.875Z INFO os_utils Trying to retrieve the remote host ###.###.###.### python path
YYYY-MM-DDTHH:MM:SS.359Z ERROR upgrade_commands Pre-upgrade checks failed. Check upgrade-source-requirements.log log for details.
YYYY-MM-DDTHH:MM:SS.58Z ERROR apply_networking Failed to get source system network configuration stdout: , stderr: eth0: error fetching interface information: Device not found, exit-code: 1.
YYYY-MM-DDTHH:MM:SS.58Z ERROR upgrade_commands Unable to execute pre-upgrade checks on host ###.###.###.### Traceback (most recent call last):

File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py", line 2189, in execute preupgradeResult = self._executePreupgradeChecks()
File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py", line 2639, in _executePreupgradeChecks srcIpv4Address, srcIpv4SubnetMask, srcIpv6Address, srcIpv6Prefix = retrieveNetworkingConfiguration(self.opsManager)
File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/transfer_network.py", line 1309, in retrieveNetworkingConfigurationinterface)
File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/apply_networking.py", line 146, in _retrieveNetworkIdentity ifconfig = _execNetworkConfigCommand(processManager, ifconfigCmd)
File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/apply_networking.py", line 71, in _execNetworkConfigCommand raise NetworkConfigError(error)NetworkConfigError: Failed to get source system network configuration stdout: , stderr: eth0: error fetching interface information: Device not found, exit-code: 1.
   
• On the source vCenter Server Appliance, running the command ifconfig does not return information for eth0.
• On the source vCenter Server Appliance, running the command less /etc/udev/rules.d/70-persistent-net.rules returns similar to:
  
  # PCI device 0x15ad:0x07b0 (vmxnet3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="##:##:##:##:##:##", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x15ad:0x07b0 (vmxnet3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="##:##:##:##:##:##", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
  
  where ATTR{address}=="##:##:##:##:##:##" displays the mac address.

Environment

vCenter Server Appliance 6.5.x

Cause

This issue occurs when the eth0 network adapter is not found on the source vCenter Server Appliance. There are multiple reasons why the eth0 adapter name could have changed such as:

• If the adapter was removed and added to the source vCenter Appliance, it will create eth1 instead of replacing eth0.
• MAC address changes will cause the adapter to be renamed.
• In some cases, if the vCenter Server appliance was restored from backup, the adapter name will change.
• The vCenter Server Appliance was cloned.

Resolution

To resolve, use one of these options:

 

Option 1: Remove the invalid entries from the 70-persistent-net.rules file

1. Connect the source vCenter Server Appliance with root credentials.
2. Run this command to enable access the Bash shell:
   
   shell.set --enabled true
    
3. Type shell and press Enter.
4. Edit the /etc/udev/rules.d/70-persistent-net.rules with this command:
   
   vi /etc/udev/rules.d/70-persistent-net.rules
    
5. Using the entries in the symptoms section edit the 70-persistent-net.rules file changing the MAC address for eth0 to the entry for eth1:
   
   For example, the current entry is:
   
   # PCI device 0x15ad:0x07b0 (vmxnet3)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="##:##:##:##:##:##", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
   
   Edit this to look similar to:
   
   # PCI device 0x15ad:0x07b0 (vmxnet3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="##:##:##:##:##:##", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

   where ATTR{address}=="##:##:##:##:##:##" is the mac address.
    
6. Restart the vCenter Server Appliance.

Option 2: Rename 70-persistent-net.rules file

1. Connect the source vCenter Server Appliance with root credentials.
2. Run this command to enable access the Bash shell:
   
   shell.set --enabled true
    
3. Type shell and press Enter.
4. Rename the 70-persistent-net.rules to 70-persistent-net.rules.old with this command:
   
   mv /etc/udev/rules.d/70-persistent-net.rules /etc/udev/rules.d/70-persistent-net.rules.old
    
5. Restart the vCenter Server Appliance.
6. Verify the /etc/udev/rules.d/70-persistent-net.rules file has been updated with the correct MAC address for eth0.

Option 3: Change the ifcfg-eth0 file to configure the current IP address

1. Open the /etc/sysconfig/network/ifcfg-eth0 file using a text editor.
2. Change the required fields such as IP address, subnet mask.
3. Restart the network service by running this command.
   
   /etc/init.d/network restart
    
4. Run ifconfig to confirm that both network adapter has the configured settings.