vCenter RDU/NDU Upgrade Fails With "Failed to initialize replication of the vCenter Server Appliance for the upgrade process"

Products

• VMware vCenter Server

Issue/Introduction

Reduced Downtime Upgrade failed with error: Failed to initialize replication of the vCenter Server Appliance for the upgrade process

The upgrade may additionally fail at the switchover step with timeout messages, similar entries will be observed in /var/log/vmware/vlcm/vlcm.log:

Wait 'NetConfig Swap' failed, reason: timeout
wfengine: execute [<###:A> netSwap.Complete(nil)] error: timeout
The activity [<###:A> netSwap.Complete(nil)] failed with non-VAPI error: timeout

Environment

VMware vCenter Server

Cause

This issue occurs when the source vCenter Server has firewall rules that block communication from unknown or temporary IP addresses.
During an RDU upgrade, the installer deploys a temporary vCenter appliance using a temporary IP address. The source vCenter must communicate with this appliance over required ports (including 443, 22 etc) to complete replication and switchover.

If the vCenter firewall is configured to allow communication only from specific IP ranges, the temporary upgrade appliance may not be able to communicate back to the existing vCenter Server.

As a result:

• Replication initialization fails, or
• Network switchover during the RDU workflow times out.

Resolution

1. Take a powered-on snapshot of the existing vCenter Server without memory and without quiescing.
   
   
2. Remove the DNS record for the Temporary vCenter I.P if any exists on the DNS server.
   
   
3. Log in to the Source vCenter Server as root while using ssh client and run the following commands to clear dns cache:

   • systemctl restart dnsmasq 

   • systemctl restart systemd-resolved

4. Switch from Bash Shell to Appliance Shell following KB: Toggling the vCenter Server Appliance default shell
   
   
5. List the current inbound firewall rules using API mentioned below while on appliance shell and save the output to notepad or take a snippet for future reference is these needs to be re-added:

   • com.vmware.appliance.version1.networking.firewall.addr.inbound.list

6. Delete all the inbound firewall rules using API mentioned below while on appliance shell:

   • com.vmware.appliance.version1.networking.firewall.addr.inbound.delete --all true

7. Re-run the RDU upgrade
   
   
8. If required add the inbound firewall rules one by one with entries collected in Step 5 using VAMI and the steps are mentioned here: Edit vCenter Firewall Settings

Additional Information

Japanese KB: vCenter の RDU/NDU アップグレードが "Failed to initialize replication of the vCenter Server Appliance for the upgrade process" で失敗する