"Failed to get exclusive lock" error while starting vpxd service

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

The VMware vCenter Server service starts then stops
In the vpxd.log file you see:

YYYY-MM-DDTHH:MM:SS info vpxd[7F6EBFAFB800] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 11000 transactions for 11 GB RAM dedicated to vpxd. YYYY-MM-DDTHH:MM:SS info vpxd[7F6EBFAFB800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc YYYY-MM-DDTHH:MM:SS info vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB YYYY-MM-DDTHH:MM:SS warning vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] [Vdb::LockRepositoryHelper] Failed to get exclusive lock: P0001:7:ODBC error: (P0001) - ERROR: Failed Lock; --> Error while executing the query YYYY-MM-DDTHH:MM:SS error vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] Unable to get exclusive access to vCenter repository. Please check if another vCenter instance is running against the same database schema. YYYY-MM-DDTHH:MM:SS warning vpxd[7F6EBFAFB800] [Originator@6876 sub=VpxMutex] Mutex VpxdVdb::InstanceMutex locked for 150202 ms YYYY-MM-DDTHH:MM:SS error vpxd[7F6EBFAFB800] [Originator@6876 sub=Main] Init failed. SystemError: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError) --> [context]zKq7AVECAAAAAFoZ0wAKdnB4ZAAATHorbGlidm1hY29yZS5zbwAAHiQbAD5yGAE48VR2cHhkAAFN81QBZktVAR0ZVQGKvFMCcAUCbGliYy5zby42AAGVtFM=[/context] YYYY-MM-DDTHH:MM:SS warning vpxd[7F6EBFAFB800] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 150306 ms YYYY-MM-DDTHH:MM:SS error vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down YYYY-MM-DDTHH:MM:SS info vpxd[7F6EBFAFB800] [Originator@6876 sub=SupportMgr] Wrote uptime information YYYY-MM-DDTHH:MM:SS info vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now

In the vmon log file you see:

YYYY-MM-DDTHH:MM:SS 198242+00:00 notice vmon Constructed command: /usr/bin/python /usr/bin/rbd-vmon-healthstatus.py YYYY-MM-DDTHH:MM:SS 392917+00:00 notice vmon Successfully executed service batch operation API_HEALTH. YYYY-MM-DDTHH:MM:SS 135743+00:00 warning vmon Service vpxd exited. Exit code 1 YYYY-MM-DDTHH:MM:SS 150980+00:00 warning vmon Service vpxd exited unexpectedly. Crash count 2. Taking configured recovery action. YYYY-MM-DDTHH:MM:SS 151318+00:00 notice vmon Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata vpxd,UNHEALTHY,HEALTHY,1

In the vpostgres log file, you see:

YYYY-MM-DDTHH:MM:SS 5d003497.287a 496944073 VCDB vc LOG: duration: 4395.807 ms statement: BEGIN;declare "SQL_CUR0x7f2d58575f90" cursor with hold for select l_stats_rollup2_proc();fetch 1024 in "SQL_CUR0x7f2d58575f90" 2019-06-12 01:45:04.601 UTC 5cf88aca.52b 0 LOG: Updating instance status... YYYY-MM-DDTHH:MM:SS 5cf88aca.52b 0 LOG: Memory check: flag = rss, mem_used_bytes = 72964 kB, mem_avail = 2890752 kB YYYY-MM-DDTHH:MM:SS 5cf88aca.52b 0 LOG: Writing instance status... YYYY-MM-DDTHH:MM:SS 5cf88aca.52b 0 LOG: Wrote instance status successfully. YYYY-MM-DDTHH:MM:SS 5cf88aca.52b 0 LOG: Updated instance status successfully. YYYY-MM-DDTHH:MM:SS 5d005937.10e4 0 VCDB vc ERROR: Failed Lock YYYY-MM-DDTHH:MM:SS 5d005937.10e4 0 VCDB vc DETAIL: Failed to exclusively lock vCenter Repository!! YYYY-MM-DDTHH:MM:SS 5d005937.10e4 0 VCDB vc STATEMENT: BEGIN;declare "SQL_CUR0x7feda8092b00" cursor with hold for SELECT SESSIONLOCK_FUNC();fetch 1024 in "SQL_CUR0x7feda8092b00" YYYY-MM-DDTHH:MM:SS 5cf88aca.52b 0 LOG: Updating instance status... YYYY-MM-DDTHH:MM:SS 5cf88aca.52b 0 LOG: Memory check: flag = rss, mem_used_bytes = 72972 kB, mem_avail = 2890752 kB

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.5.x

Cause

This issue will occurred when the 2 concurrent vpxd processes try to acquire the lock of the db schema.

Resolution

This issue occurs when the 2 concurrent processes were trying to acquire the lock of the DB schema.
vpxd unable to start, this issue is widely observed when the lock on database has already held by vpxd.

Note: There are a couple of more manual vpxd restarts in the next 10 or so minutes. While vpxd was getting restarted manually, vmon was also trying to get the service back on.

As per the above log proof, while vMon was trying to start the vmware-vpxd service, another manual start of /etc/init.d/vmware-vpxd was issued.


Validate the vpxd service status, if the service already started, stop the service and start the service using the command

service-control --stop --all && service-control --start --all

Additional Information

vCenter Server does not start when connecting to an Oracle database with the vpxd.log error: Unable to get exclusive access to VC repository (broadcom.com)