"Failed to reconcile ifaces network setting" while upgrading Supervisor Cluster

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• Upgrading Supervisor Cluster fails with below error:
  • Configured Supervisor Control plane VM's Workload Network Configuration error (since MM/DD/YY, HH:MM:SS PM) System error occurred on Master node with identifier <node_id>. Details: Failed to reconcile ifaces network setting. Details Timed out waiting for ifaces to come up ..

• Supervisor Control Plane VM node has management address and lost workload address even both vNICs are connected in the VM's edit settings.
  
  
• On the failed Supervisor Control Plane node, eth1 shows down.
  • # ip link show
1: lo: <LOOPBACK, UP, LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BRDADCAST, NULTICAST, UP, LOWER UP> mtu 1500 qdisc mq state UP node DEFAULT group default qlen 1089
link/ether <ID> brd ff:ff:ff:ff:ff:ff
altnane eno1
altname enp11se
altname ens192
3: eth1: <NO-CARRIER, BROADCAST, MULTICAST, UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether <ID> brd ff:ff:ff:ff:ff:ff
altnane enoz
altname enp1953
altnane ens224

  • # networkctl
IOX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier      unmanaged
2 eth0 ether  routable    contigured
3 eth1 ether  no-carrier  configuring

    3 links listed.

  • However, the eth1 has the IP address allocated
    • cat /etc/systemd/network/10-ethl.network

    • [Match]
Name = eth1

      [Network]
IPv6AcceptRA = no
Address = x.x.x.x/x
DNS - x.x.x.x
Domains = <domain-name>

Cause

Newly deployed Supervisor VM are provisioned on the faulty ESXi host which shows an error on NSX 

Resolution

• Workaround 1:
  • Put the problematic ESXi host on Maintenance mode to continue with the Supervisor Upgrade.
  • Once the newly deployed Supervisor Control Plane VMs are moved to different ESXi host, it should show eth1 UP and show IP address on the VM's Summary page.
  • On Supervisor UI you should be able to see the upgrade progress.
    
    
• Workaround 2:
  • To address the problem ESXi host, follow the NSX KB