Error: couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused

Products

• Tanzu Kubernetes Grid

Issue/Introduction

      Following symptoms shown:

• kubectl command results in error:
  couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
  
  
• The Kubernetes API server container is in an "exited" state.
• etcd cluster doesn't show any members
• Attempts to connect to the KubeVIP are unsuccessful.

      How to Check:

1. SSH into the host
   ssh capv@<managment node ip>
2. Find the etcdctl executable:
   find /var -name etcdctl
3. Create an alias for etcdctl
   alias etcdctl=<path from output of find command in step 2>
4. Get members of the ETCD cluster, which doesn't show any members
   etcdctl member list
5. "crictl ps -a" command shows kube-apiserver container in exited status
   # crictl ps -a
CONTAINER       IMAGE       CREATED      STATE            NAME            ATTEMPT             POD ID              POD
#####           #####       <time>       Exited           kube-apiserver  ####                ####                                <pod name>

Environment

Tanzu Kubernetes Grid 

Cause

A mismatch was identified between the IPs in the etcd.yaml file and the IPs assigned to the control plane VMs. The IPs originally assigned to the control plane nodes were changed, which caused a loss of the etcd quorum.

The cloud-init output log (cat /var/log/cloud-init-output.log) to confirm the original IPs assigned to the control plane manager vm differ from current ips shown in vcenter server > inventory vm > summary > ip address.

output of kubectl get nodes -o wide showed same result and the etcd.yaml file located at "/etc/kubernetes/manifests/etcd.yaml" under the "--listen-peer-urls" field reflected these same mismatched IPs.

The mismatch between the assigned IPs and the original IPs of the VMs caused the etcd quorum to be lost.

The ip change can happen when DHCP leases are not statically assigned to control plane vms during provisioning.

Resolution

1. Reassign the original IPs to the control plane VMs

2. Update the DHCP server to reflect the correct IP and lease time assignments

or

3. Manually update/etc/systemd/network/<network file> on the affected Control Plane VMs (for Photon VM):

DHCP=no
address=<noted address from etcd.yaml>
gateway=<gateway address>

4. Restart network using:
    systemctl restart systemd-networkd

5. Confirm IP address is reflected correctly on all management Control Plane VMs 

6. Confirm etcd is restarted using command from (above step 4 on issue): etcdctl member list

7. Confirm kube-apiserver as well using crictl ps

Once etcd and kube apiserver are started other services should also begin to startup.

Additional Information

For more information on manual ip update see Setting a static ip address

Information regarding kubernetes etcd see Configure-upgrade-etcd