vpxd.log flooded with messages: "N20DistEsxManagerClient11OpExceptionE(Timeout calling 'runReplica')"

Products

• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x
• VMware vSphere ESXi
• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

vpxd.log has the following messages:

Failed to run replica; [vim.HostSystem:host-###,<FQDN of ESXi>], N20DistEsxManagerClient11OpExceptionE(Timeout calling 'runReplica')

hostd.log in ESXi has:

Admin command failed; '/usr/lib/vmware/clusterAgent/bin/clusterAdmin ++group=hostd-tmp,mem=30 cluster runReplica ########-####-####-####-########:domain-c#### <FQDN of ESXi>:2379 <FQDN of another ESXi> --hostname <FQDN of another ESXi> --opid CSMM-domain-c####-###-####', exit: 2, stderr:

Environment

vSphere 8.x

Cause

FQDN name resolution for ESXi in the cluster is not configured properly.

/var/run/log/clusterAgent.log

YYYY-MM-DDThh:mm:ssZ No(5) clusterAgent[#######]: INFO  Replacing opID  {"opID": "adminsrv-####", "new": "CSMM-domain-c####-###-####-####"}
YYYY-MM-DDThh:mm:ssZ No(5) clusterAgent[#######]: INFO  Processing admin request    {"opID": "CSMM-domain-c####-###-####-####", "type": "TYPE_RUN_REPLICA", "request": {"type":5,"op_id":"CSMM-domain-c####-###-####","run_replica":{"cluster_id":"########-####-####-####-################:domain-c####","api_addresses":["<FQDN of another ESXi>:2379"],"hostname":"<FQDN of ESXi>","username":"<FQDN of ESXi>","password":"******"}}}
YYYY-MM-DDThh:mm:ssZ No(5) clusterAgent[#######]: INFO  Supervisor handling admin request   {"opID": "CSMM-domain-c####-###-####-####"}
YYYY-MM-DDThh:mm:ssZ No(5) clusterAgent[#######]: INFO  Creating etcd client wrapper    {"opID": "CSMM-domain-c####-###-####-####", "wrapper": "0xc0001fd3c0"}
YYYY-MM-DDThh:mm:ssZ No(5) clusterAgent[#######]: INFO  Creating etcd client    {"opID": "CSMM-domain-c####-###-####-####", "endpoints": ["<FQDN of another ESXi>:2379"], "tls": true, "username": "<FQDN of ESXi>", "timeout": "5s", "incomplete-password-refresh": false}
YYYY-MM-DDThh:mm:ssZ No(5) clusterAgent[#######]: WARN  grpc: addrConn.createTransport failed to connect to {<FQDN of another ESXi>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup <FQDN of another ESXi>: no such host". Reconnecting...
YYYY-MM-DDThh:mm:ssZ No(5) clusterAgent[#######]: WARN  grpc: addrConn.createTransport failed to connect to {<FQDN of another ESXi>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup <FQDN of another ESXi>: no such host". Reconnecting...
YYYY-MM-DDThh:mm:ssZ No(5) clusterAgent[#######]: WARN  grpc: addrConn.createTransport failed to connect to {<FQDN of another ESXi>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup <FQDN of another ESXi>: no such host". Reconnecting...
YYYY-MM-DDThh:mm:ssZ No(5) clusterAgent[#######]: ERROR Failed to create new etcd client    {"opID": "CSMM-domain-c####-###-####-####", "error": "context deadline exceeded", "error value": "context.deadlineExceededError{}"}

Resolution

Check and update DNS server setting of ESXi.

Refer "Configuring DNS settings" in Configuring VMware ESXi Management Network from the direct console.