vCenter vpostgres service hang due to Poor NFS storage performance

Products

• VMware vCenter Server

Issue/Introduction

• vCenter is not accessible.
• Logging in to the vCenter Server using the vSphere Client fails.
• The following error is observed when trying to access the vCenter Server from a web browser:

no healthy upstream

• The time taken for vCenter database tasks increased significantly, reaching over 30 seconds：

VCDB vpxd [local] xxx 4LOG:  duration: 23789.467 ms  statement: COMMIT
VCDB vpxd [local] xxx 4LOG:  duration: 4068.737 ms  statement: COMMIT
VCDB vpxd [local] xxx 6LOG:  duration: 31925.616 ms  statement: COMMIT

• vpostgres service is not responding:

vmon.log:

Wa(03) host-xxxx <vmware-vpostgres-healthcmd> SysProcess exec timed out. Force kill. Pid xxxx
Wa(03) host-xxxx <vmware-vpostgres> Service api healthcheck command timedout.
Er(02) host-xxxx <vmware-vpostgres> health state unknown ,considered as system failure

 

Environment

VMware vCenter Server

Cause

NFS datastore I/O latency increased, then caused vCenter vpostgres service not responding, several other services are impacted.

NFS datastore performance deteriorated. I/O latency increased:

vobd.log

Wa(180) vmkwarning: cpuxx:xxxx)WARNING: NFS: xxxx: NFS volume <datastore_name> performance has deteriorated. I/O latency increased from average value of 4xxx(us) to 19xxxxx(us)

 

Resolution

Troubleshoot and optimize storage performance.