vCenter server upgrade failed at Stage2 reaching 50% while initializing Import Data due to network connectivity failure

Products

• VMware vCenter Server

Issue/Introduction

While upgrading vCenter server, the upgrade process consistently freeze at 50% during Stage 2 during data import. On a broader scope, the issue occurrence is due to network connectivity issue. This articles guides users in validating network communication and fix issue, that occurred due to configuration set in vDS. It also mentions about other aspects to consider with a quick check to avoid upgrade failure at stage 2.

Cause

Network security policy restrictions (Specifically, L2 security violations on the DVS port group blocking traffic between the source and destination subnets.The failure at 50% occur during "Data transfer" phase, which requires stable network connectivity between the source and destination VCSA. The `net-dvs -l port` output confirmed `com.vmware.common.port.volatile.status = inUse linkUp blocked portID=####`, validating that the network traffic was being dropped by the security policy at the switch level. 

 

• # net-dvs -l

  port ###:
com.vmware.common.port.alias =  ,propType = CONFIG
com.vmware.common.port.connectid = ####,  propType = CONFIG
com.vmware.common.port.volatile.status = inUse linkUp blocked portID=#### Port blocked due to L2 security violation propType = RUNTIME
 com.vmware.common.port.volatile.vlan = VLAN ##
 propType = RUNTIME
 com.vmware.common.port.portgroupid = dvportgroup-#### , propType = CONFIG
 com.vmware.common.port.block = false ,  propType = CONFIG
 com.vmware.common.port.dvfilter = filters (num = 0):
 propType = CONFIG
 com.vmware.vswitch.port.teaming.policy:
 load balancing = source virtual port id
 link selection = link state up;
 link behavior = notify switch; best effort on failure; shotgun on failure;
 active = Uplink 1, Uplink 2,
 standby =
 propType = CONFIG
 com.vmware.vswitch.port.security = deny promiscuous; deny mac change; deny forged frames
 propType = CONFIG
 com.vmware.vswitch.port.macManagement:
 Allow MAC Change = False
 MAC Learning = False
 Unknown Unicast Flooding = False
 MAC Limit = 4096
  MAC Limit Policy = ALLOW
 propType = CONFIG

Resolution

1. Ensure both forward and reverse DNS lookup for the VCSA FQDN resolve correctly.
2. Validate network communication in terms of configured subnet and its associated Gateway address. 
3. Update Security Policy by following below mentioned steps:
   Enable Promiscuous mode and Enable Mac change in specific portgroup of Distributed switch.
1. Log in to the vSphere Client.
2. Navigate to Networking: Click the Networking icon in the left-hand navigation pane to view your network topology.
3. Locate the Port Group: Expand your Distributed Switch (VDS) and find the specific Distributed Port Group you want to configure.
4. Edit Settings: Right-click the target Distributed Port Group and select Edit Settings.
5. Modify Security Policy:
1. Click on the Security option in the left menu of the settings window.
2. Locate Promiscuous mode and select Accept from the drop-down menu.
3. Locate MAC address changes and select Accept from the drop-down menu.
6. Save Changes: Click OK to apply and save the configuration changes.
   
   
4. Firewall Validation: Confirm that migration ports (TCP 9123) and management ports (SSH 22, HTTPS 443, VAMI 5480) are permitted through any intermediate firewalls.

Additional Information

• Ping testing from the jump box failed due to difference in IP subnet.
• Ping testing from source ESXi host to the destination VCSA temporary IP resulted in 100% packet loss as Gateway was unreachable. 
• Upgrade logs (specifically `/var/log/vmware/upgrade/upgrade_runner.log`) indicated network connectivity timeouts.
• Distributed Virtual Switch (DVS) port validation showed a "Port blocked due to L2 security violation" error on the destination port group.