VMware Cloud Director intermittently logs users out with lock timeout errors in the logs

Products

• VMware Cloud Director

Issue/Introduction

VMware Cloud Director is intermittently logging users out during multiple requests. In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log we see errors similar to the following:

2025-03-24 15:33:01,588 | ERROR    | pool-jetty-15928          | JDBCExceptionReporter          | ERROR: canceling statement due to lock timeout
  Where: while rechecking updated tuple (212,76) in relation "user_session" | requestId=#######################,request=GET https://<your VCD FQDN>.com/cloudapi/1.0.0/orgs/urn:vcloud:org:############/quotas,requestTime=1742830348376,remoteAddress=##########:54566,userAgent=vcd-openapi-client,accept=application/json;version 38.1

Environment

VMware Cloud Director 10.x

Cause

There are two possible causes:

1. The VMware Cloud Director cells are undersized or another cell is required for the pool. Currently the cells are unable to handle a multitude of requests.
2. The underlying infrastructure is having issues that are causing performance degradation.

Resolution

If the appliances are undersized or the cell count is under 6 then the VMware Cloud Director Appliance Sizing Guidelines should be followed. 

At the same time check for the following on the management ESXi host cluster:

• Disk I/O latency.
• CPU and Memory contention.
• Physical disk and networking issues.

This problem has occurred due to physical disk latency issues prior, and it can be resolved by migrating management VMs off of the issue ESXi host and placing the ESXi host in maintenance mode while it is corrected.