etcd service configuration file is not updated after ESXi changing mgmt ip address

Products

• VMware vSphere ESXi

Issue/Introduction

• Due to data center migration, the management IP addresses of all ESXi hosts in the cluster are changed.
• After mgmt ip changing, etcd service configuration file ./scratch/var/clusterAgent/etcd.yml is not auto updated, the saved ip address is not current ESXi mgmt ip.

name: ########################
data-dir: /var/cache/datafiles/etcdData
initial-cluster:#########################=https://C.C.C.C:2380  <<<<< This ip address is not ESXi current mgmt ip.
initial-cluster-state: new
listen-peer-urls: https://0.0.0.0:2380
initial-advertise-peer-urls: https://C.C.C.C:2380
advertise-client-urls: https://C.C.C.C:2379
listen-client-urls: https://0.0.0.0:2379
...

IP address C.C.C.C is this ESXi's previous mgmt ip address, not current mgmt ip.

• The etcd cluster still use ESXi old mgmt ip addresses A.A.A.A, B.B.B.B and C.C.C.C. These three IPs are not in use.

clusterAgent.log:
No(5) clusterAgent[2099412]: INFO  Creating etcd client    {"opID": "supervisor-####", "endpoints": ["A.A.A.A:2379", "B.B.B.B:2379", "C.C.C.C:2379"], "tls": true, "username": "C.C.C.C", "timeout": "30s", "incomplete-password-refresh": false}
No(5) clusterAgent[2099412]: WARN  grpc: addrConn.createTransport failed to connect to {C.C.C.C:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcpC.C.C.C:2379: i/o timeout". Reconnecting...
No(5) clusterAgent[2099412]: WARN  grpc: addrConn.createTransport failed to connect to {B.B.B.B:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcpB.B.B.B:2379: i/o timeout". Reconnecting...
No(5) clusterAgent[2099412]: WARN  grpc: addrConn.createTransport failed to connect to {A.A.A.A:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcpA.A.A.A:2379: i/o timeout". Reconnecting...

• ESXi etcd service still connecting other ESXi hosts' old mgmt ip addresses. 

etcd.log：
Er(3) etcd[2099430]: publish error: etcdserver: request timed out
Wa(4) etcd[2099430]: health check for peer aadba2c1e56f4cae could not connect: dial tcp A.A.A.A:2380: i/o timeout
Wa(4) etcd[2099430]: health check for peer 59f8bc13b00ed505 could not connect: dial tcp B.B.B.B:2380: i/o timeout
Wa(4) etcd[2099430]: health check for peer 59f8bc13b00ed505 could not connect: dial tcp B.B.B.B:2380: i/o timeout

Environment

VMware vSphere ESXi

Resolution

Reinstall ESXi host fixed this issue.

Additional Information

Reference KB: https://knowledge.broadcom.com/external/article?articleNumber=385110