"Cannot run /sbin/ifup eth0 command. Unknown error. Return code : 256 output: Make sure the interface is down or not assigned any IP" error happens when upgrading vCenter

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Upgrade of vCenter Server Appliance to 8.0 U3 fails at 50% at setting network.
• In /var/log/vmware/deploy/vminst-con.log or /var/log/vmware/upgrade/bootstrap.log file, you see entries similar to:

2024-08-19T07:15:03.197Z  Executing command: ['/opt/vmware/share/vami/vami_set_network', 'eth0', 'STATICV4', '10.x.x.50', '255.255.255.128', '10.x.x.1']
2024-08-19T07:15:03.197Z  Executing command: ['/opt/vmware/share/vami/vami_set_network', 'eth0', 'STATICV4', '10.x.x.50', '255.255.255.128', '10.x.x.1'] with retryCount: 5, waitTimeInSecs: 15
2024-08-19T07:15:03.197Z  Running command: ['/opt/vmware/share/vami/vami_set_network', 'eth0', 'STATICV4', '10.x.x.50', '255.255.255.128', '10.x.x.1']
2024-08-19T07:15:03.400Z  Done running command
2024-08-19T07:15:03.400Z  ERROR: Command failed with exit status 1
2024-08-19T07:15:03.400Z  Stdout: net.ipv6.conf.eth0.disable_ipv6 = 1

2024-08-19T07:15:03.400Z  Stderr: 19/08/2024 07:15:03 [ERROR] Cannot run /sbin/ifup eth0 command. Unknown error. Return code : 256 output: Make sure the interface is down or not assigned any IP
eth0 is DOWN or not assigned an IP. Bringnig eth0 up...
Can not find the manual filename, let us search for the auto filename
Performing duplicate address check for IPv4 address 10.x.x.50
^[[1;31mError: IP already exists in the network^[[0;39m

Unable to set the network parameters

2024-08-19T07:15:03.400Z  Command failed with exit status 1. Detail: 19/08/2024 07:15:03 [ERROR] Cannot run /sbin/ifup eth0 command. Unknown error. Return code : 256 output: Make sure the interface is down or not assigned any IP
eth0 is DOWN or not assigned an IP. Bringnig eth0 up...
Can not find the manual filename, let us search for the auto filename
Performing duplicate address check for IPv4 address 10.x.x.50
^[[1;31mError: IP already exists in the network^[[0;39m

Unable to set the network parameters

2024-08-19T07:15:03.400Z  Retries remaining: 4

Environment

VMware vCenter Server Appliance 7.0.x
VMware vCenter Server Appliance 8.0.x

Cause

This issue occurs when: 

• The source vCenter is shut down.
• The target vCenter uses the Address Resolution Protocol (ARP) query to detect the liveness of the source vCenter IP.
• The physical switch answers ARP queries for the source vCenter's IP address.

Resolution

To resolve this issue, let the target vCenter retry more times until the ARP cache of the source vCenter IP expires in the physical switches.

• 1. After stage 1 of the upgrade process is done, take an SSH session to the target vCenter and run this command:

sed -i.bak -e 's/NETWORK_RETRY_COUNT = 5/NETWORK_RETRY_COUNT = 80/g' /bin/setnet

• 2. Verify that the value of NETWORK_RETRY_COUNT is changed to 80:

grep NETWORK_RETRY_COUNT /bin/setnet

• 3. Continue stage 2 of the upgrade process; the target vCenter can apply network settings.