"Unable to acquire JDBC Connection" error when accessing Fleet Management in VCF Operations

Products

• VCF Operations

Issue/Introduction

• Accessing Fleet Management from the VCF Operations UI fails and displays the following error:
  Unable to acquire JDBC Connection; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
  
  
• The VCF Operations Admin UI displays the node as connected.
  
  
• Running vrlcm-cli --health-status on the Fleet Management appliance via SSH reveals the postgres service is down.
  bootstrap: SUCCESS
postgres: Not Running
vrlcm-server: Running
blackstone: Running
  
  
  
• Manually starting the PostgreSQL service (systemctl start vpostgres.service) fails with a control process error.
  systemctl start vpostgres.service Job for vpostgres.service failed because the control process exited with error code. See "systemctl status vpostgres.service" and "journalctl -xeu vpostgres.service" for details.
  
  
  
• Log output from journalctl -xeu vpostgres.service indicates a stale postmaster.pid file on a read-only file system:
  postgres[###]: pg_ctl: another server might be running; trying to start server anyway
postgres[###]: ###-##-## ##:##:### UTC [####] FATAL:  could not remove old lock file "postmaster.pid": Read-only file system
postgres[###]: ##-##-## ##:##:### UTC [###] HINT:  The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again.
postgres[####]: pg_ctl: could not start server
postgres[####]: Examine the log output.
systemd[1]: vpostgres.service: Control process exited, code=exited, status=1/FAILURE
  
  
  

Cause

• Fleet management appliance file system transitions to a read-only state to protect against underlying storage anomalies or disconnects.
• Due to the locked file system, the PostgreSQL service (vpostgres.service) cannot delete a stale postmaster.pid lock file during startup. This failure prevents the database engine from initializing, breaking the JDBC connection required to load the Fleet Management interface.

Resolution

Manually rename the corrupted database lock file to allow the PostgreSQL service to initialize successfully.

Prerequisite Take an offline (powered-off) snapshot of  Fleet Management virtual machine before proceeding.

1. SSH session to Fleet Management appliance and log in as the root user.
   
   
2. Verify the presence of the postmaster.pid file in the database directory:
   ls -l /storage/db/pgdata/
   
   
   
3. Rename the stale lock file to remove the active lock while retaining a backup:
   mv /storage/db/pgdata/postmaster.pid /storage/db/pgdata/postmaster.pid_bkp
   
   
   
4. Start the PostgreSQL service:
   systemctl start vpostgres.service

Note: The service automatically generates a new PID file upon initialization.



5. Verify the health status of the Fleet Management services:
   vrlcm-cli --health-status
   
   

   Expected Output:
   bootstrap: SUCCESS
postgres: Running
vrlcm-server: Running
blackstone: Running
   
   

6. Log in to the VCF Operations UI to confirm the Fleet Management interface loads without the JDBC exception.