kubectl
command
couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
s results in error:
ssh capv@<managment node ip>
find /var -name etcdctl
alias etcdctl=<path from output of find command in step 2>
etcdctl member list
[ ~ ]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
##### ##### <time> Exited kube-apiserver #### ####
<pod name>
2.x
A mismatch was identified between the IPs in the etcd.yaml file and the IPs assigned to the control plane VMs. The IPs originally assigned to the control plane nodes were changed, which caused a loss of the etcd quorum.
The cloud-init output log (cat /var/log/cloud-init-output.log) to confirm the original IPs assigned to the control plane manager vm differ from current ips shown in vcenter server > inventory vm > summary > ip address.
output of kubectl get nodes -o wide showed same result and the etcd.yaml file located at "/etc/kubernetes/manifests/etcd.yaml" under the "--listen-peer-urls" field reflected these same mismatched IPs.
The mismatch between the assigned IPs and the original IPs of the VMs caused the etcd quorum to be lost.
The ip change can happen when DHCP leases are not statically assigned to control plane vms during provisioning.
1. Reassign the original IPs to the control plane VMs
2. Update the DHCP server to reflect the correct IP and lease time assignments
or
3. Manually update on the vm (if a photon deployed control plane vm):
a. vi
command on /etc/systemd/network/<network file>
and press i
to update the file with the following parameters
i. DHCP=no
ii. address=<noted address from etcd.yaml>
iii. gateway=<gateway address
>
iv. confirm settings and saved using :wq!
4. Restart network using:
systemctl restart networkd-systemd
5. Confirm Ip address is reflected correctly on all management control plane vms
6. Confirm etcd is restarted using command from (above step 4 on issue): etcdctl member list
7. Confirm kube-apiserver as well using crictl ps
Once etcd and kube apiserver are started other services should also begin to startup.
For more information on manual ip update see Setting a static ip address
Information regarding kubernetes etcd see Configure-upgrade-etcd