During Deployment/Upgrade of a Supervisor Cluster the Supervisor Control Plane VM shows errors and does not progress

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

During deployment of a Supervisor Cluster, the Supervisor is showing errors and deployments do not progress.

The following error messages are seen in the Supervisor:

Configured Control Plane VMs
Configuration error (since #/#/####, ##:##:## AM)
System error occurred on Master node with identifier ################################. Details: Failed to sync changes: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'cm', '###-###-######', '--namespace', 'vmware-system-nsx', '-o', 'yaml']' returned ###-#### exit status 1.. Will be retried..

Customized guest of Supervisor Control plane VM
Configuration error (since #/#/####, ##:##:## AM)
System error occurred on Master node with identifier ################################. Details: Log forwarding sync update failed: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'configmap', '#########-######-######', '--namespace', 'vmware-system-logging', '--######-###-#####=true', '-o', 'json']' returned ###-#### exit status 1..

Configured Supervisor Control plane VM's Workload Network
Configuration error (since #/#/####, ##:##:## AM)
System error occurred on Master node with identifier ################################. Details: Failed to sync changes: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'cm', '###-###-######', '--namespace', 'vmware-system-nsx', '-o', 'yaml']' returned ###-#### exit status 1.. Will be retried..

Environment

vSphere Supervisor

Cause

The k8s-admin-conf file is corrupt on the affected Supervisor Control Plane VM

Resolution

Replace the k8s-admin-conf file on the faulty Supervisor Control Plane VM with the k8s-admin-conf file from a functioning Supervisor Control Plane VM

1. Backup the k8s-admin-conf file on the faulty Supervisor Control Plane VM

mv /dev/shm/wcp_decrypted_data/k8s-admin-conf /dev/shm/wcp_decrypted_data/k8s-admin-conf.bak

2. Copy the k8s-admin-conf from a working Supervisor Control Plane VM to the faulty Supervisor Control Plane VM

Run the following command from a working Supervisor Control Plane VM

scp /dev/shm/wcp_decrypted_data/k8s-admin-conf root@affected_CPVM_IP:/dev/shm/wcp_decrypted_data/