Supervisor in Configuration error after restoring vCenter from VAMI file base Backup

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

 

• vCenter restored successfully from VAMI backup.

• Supervisor Cluster shows configuration error:

  System error occurred on Master node with identifier.
  Details: Failed to reconcile ifaces network setting. Timed out waiting for ifaces to come up.
  

• Logs from wcpsvc.log indicate connectivity and configuration errors:

  YYYY-MM-MM error wcp [licensemonitor/license_event_monitor.go:XXX] [opID=licenseRefreshMonitor] Supervisor control plane failed: No connectivity to API Master: connectivity ok, config status ERROR
  YYYY-MM-MM debug wcp [cnslib/resourcecheck_task_utils.go:208] [opID=XXXXX] Retrying resource check
  YYYY-MM-MM warning wcp [kubelib/retry.go:93] [opID=workload_sync-8c8c-8ca0] Request to apiserver failed. Err <nil>, Endpoint http://localhost:1080/external-cert/http1/VIP IP/6443/apis/vmoperator.vmware.com/v1alpha1/namespaces/ns01/virtualmachineclasses?timeout=2m0s. Will be retried.

• Supervisor control plane nodes are in ready state, Command to check - kubectl get nodes -A

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:a3:20:e8 brd ff:ff:ff:ff:ff:ff
    altname eno1
    altname enp11s0
    altname ens192
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:50:56:a3:25:ce brd ff:ff:ff:ff:ff:ff
    altname eno2
    altname enp19s0
    altname ens224

# networkctl
IDX LINK TYPE     OPERATIONAL SETUP
  1 lo   loopback carrier     unmanaged
  2 eth0 ether    routable    configured
  3 eth1 ether    no-carrier  configuring

• Check if the network adapter is connected On Supervisor control plane VM

 

 

Environment

vSphere with Tanzu

vCenter Server Appliance

Cause

During vCenter restoration from backup, ESXi hosts and vCenter became out of sync.

Network interfaces on Supervisor Control Plane VMs were disconnected, causing the communication failures.

Resolution

• Step 1: Reconnect Supervisor Control Plane Network Adapters
  Create a temporary user for vCenter administrative access using the article - Bypassing vSphere with Tanzu managed virtual machine permissions for troubleshooting purposes and Attempt to reconnect network adapters on Supervisor Control Plane VMs. If this fails with error - "The operation is not allowed in the current state of this host" check further steps

   

• Step 2: Check on stale or hung tasks:
  Verify if there is any stale or hung Maintenance mode task on the ESXi host.

  • For any stale task is found in the task of the vCenter Cancel the task.

  • If the option is greyed out. Restart vpxd service on vCenter to clear the stale task. 

    • NOTE: Verify the task details and perform the action plan on the vCenter.
      service-control --stop vpxd && service-control --start vpxd

      Attempt to connect the network adapters on Supervisor Control Plane VMs