Supervisor cluster has a configuration error.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Following error is seen on the vCenter UI for the supervisor cluster :

Customized guest of Supervisor Control plane VM
 • Configuration error (since dd/mm/yyyy, hh:mm:ss)
 • System error occurred on Master node with identifier ############. Details: Log forwarding sync update failed: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'configmap', 'fluentbit-config-system', '--namespace', 'vmware-system-logging', '--ignore-not-found=true', '-o', 'json']' returned non-zero exit status 1..
Configured Supervisor Control plane VM's Management Network
Configured Supervisor Control plane VM as Kubernetes Control Plane Node
Configured Supervisor Control plane VM's Workload Network
 • Configuration error (since dd/mm/yyyy, hh:mm:ss)
 • System error occurred on Master node with identifier ############. Details: Timed out waiting for APIServer Pod spec to reflect changes done to manifest file.
 • System error occurred on Master node with identifier ############. Details: Nginx proxy config for Pinniped update failed: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'svc', 'pinniped-supervisor', '--namespace', 'vmware-system-pinniped', '--ignore-not-found=true', '-o', 'jsonpath={.spec.clusterIP}']' returned non-zero exit status 1..
 • System error occurred on Master node with identifier ############. Details: Nginx proxy config for authproxy update failed: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'secret', 'wcp-authproxy-client-secret', '--namespace', 'kube-system', '--ignore-not-found=true', '-o', 'json']' returned non-zero exit status 1..

• Verifying the kubelet logs on the impacted node, following errors are seen:

journalctl -xeu kubelet 

Output:

Unable to authenticate the request due to an error" err="Post \"https://127.0.0.1:6443/apis/authentication.k8s.io/v1/tokenreviews\": dial tcp 127.0.0.1:6443: connect: connection refused"

Environment

vSphere Kubernetes Service

Resolution

Restart the kubelet service on the impacted node by running the following command :

systemctl restart kubelet 

Additional Information

Japanese KB: Supervisorクラスタが構成エラーになる