"Failed to get exclusive lock" error while starting vpxd service
search cancel

"Failed to get exclusive lock" error while starting vpxd service

book

Article ID: 309408

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • The VMware vCenter Server service starts then stops
  • In the vpxd.log file you see:
2019-06-12T01:47:58.924Z info vpxd[7F6EBFAFB800] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 11000 transactions for 11 GB RAM dedicated to vpxd. 2019-06-12T01:47:58.924Z info vpxd[7F6EBFAFB800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc 2019-06-12T01:47:59.078Z info vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB 2019-06-12T01:47:59.124Z warning vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] [Vdb::LockRepositoryHelper] Failed to get exclusive lock: P0001:7:ODBC error: (P0001) - ERROR: Failed Lock; --> Error while executing the query 2019-06-12T01:50:29.126Z error vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] Unable to get exclusive access to vCenter repository. Please check if another vCenter instance is running against the same database schema. 2019-06-12T01:50:29.126Z warning vpxd[7F6EBFAFB800] [Originator@6876 sub=VpxMutex] Mutex VpxdVdb::InstanceMutex locked for 150202 ms 2019-06-12T01:50:29.126Z error vpxd[7F6EBFAFB800] [Originator@6876 sub=Main] Init failed. SystemError: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError) --> [context]zKq7AVECAAAAAFoZ0wAKdnB4ZAAATHorbGlidm1hY29yZS5zbwAAHiQbAD5yGAE48VR2cHhkAAFN81QBZktVAR0ZVQGKvFMCcAUCbGliYy5zby42AAGVtFM=[/context] 2019-06-12T01:50:29.127Z warning vpxd[7F6EBFAFB800] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 150306 ms 2019-06-12T01:50:29.127Z error vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down 2019-06-12T01:50:29.127Z info vpxd[7F6EBFAFB800] [Originator@6876 sub=SupportMgr] Wrote uptime information 2019-06-12T01:50:29.127Z info vpxd[7F6EBFAFB800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now
  • In the vmon log file you see:
19-06-12T01:49:50.198242+00:00 notice vmon Constructed command: /usr/bin/python /usr/bin/rbd-vmon-healthstatus.py 19-06-12T01:49:50.392917+00:00 notice vmon Successfully executed service batch operation API_HEALTH. 19-06-12T01:50:29.135743+00:00 warning vmon Service vpxd exited. Exit code 1 19-06-12T01:50:29.150980+00:00 warning vmon Service vpxd exited unexpectedly. Crash count 2. Taking configured recovery action. 19-06-12T01:50:29.151318+00:00 notice vmon Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata vpxd,UNHEALTHY,HEALTHY,1
 
  • In the vpostgres log​​​​​​​ file, you see:
 2019-06-12 01:45:04.396 UTC 5d003497.287a 496944073 VCDB vc LOG: duration: 4395.807 ms statement: BEGIN;declare "SQL_CUR0x7f2d58575f90" cursor with hold for select l_stats_rollup2_proc();fetch 1024 in "SQL_CUR0x7f2d58575f90" 2019-06-12 01:45:04.601 UTC 5cf88aca.52b 0 LOG: Updating instance status... 2019-06-12 01:45:04.602 UTC 5cf88aca.52b 0 LOG: Memory check: flag = rss, mem_used_bytes = 72964 kB, mem_avail = 2890752 kB 2019-06-12 01:45:04.602 UTC 5cf88aca.52b 0 LOG: Writing instance status... 2019-06-12 01:45:04.602 UTC 5cf88aca.52b 0 LOG: Wrote instance status successfully. 2019-06-12 01:45:04.602 UTC 5cf88aca.52b 0 LOG: Updated instance status successfully. 2019-06-12 01:45:27.734 UTC 5d005937.10e4 0 VCDB vc ERROR: Failed Lock 2019-06-12 01:45:27.734 UTC 5d005937.10e4 0 VCDB vc DETAIL: Failed to exclusively lock vCenter Repository!! 2019-06-12 01:45:27.734 UTC 5d005937.10e4 0 VCDB vc STATEMENT: BEGIN;declare "SQL_CUR0x7feda8092b00" cursor with hold for SELECT SESSIONLOCK_FUNC();fetch 1024 in "SQL_CUR0x7feda8092b00" 2019-06-12 01:46:04.656 UTC 5cf88aca.52b 0 LOG: Updating instance status... 2019-06-12 01:46:04.656 UTC 5cf88aca.52b 0 LOG: Memory check: flag = rss, mem_used_bytes = 72972 kB, mem_avail = 2890752 kB


Environment

VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.5.x

Cause

This issue will occurred when the 2 concurrent vpxd processes try to acquire the lock of the db schema.

Resolution

This issue occurs when the 2 concurrent processes were trying to acquire the lock of the DB schema.
vpxd unable to start, this issue is widely observed when the lock on database has already held by vpxd.

Note: There are a couple of more manual vpxd restarts in the next 10 or so minutes. While vpxd was getting restarted manually, vmon was also trying to get the service back on.

As per the above log proof, while vMon was trying to start the vmware-vpxd service, another manual start of /etc/init.d/vmware-vpxd was issued.


Validate the vpxd service status, if the service already started, stop the service and start the service using the command

service-control --stop --all && service-control --start --all


Additional Information

vCenter Server does not start when connecting to an Oracle database with the vpxd.log error: Unable to get exclusive access to VC repository

Powered by Wolken Software