Downtime encountered on vCD site and may cause Veeam backup jobs to fail

Products

• VMware Cloud Director

Issue/Introduction

UI Shows error message similar to:

HTTP ERROR 404 JSP file [/error.jsp] not found
URI: /provider/
STATUS: 404
MESSAGE: JSP file [/error.jsp] not found
SERVLET: jsp
CAUSED BY: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
CAUSED BY: org.hibernate.exception.GenericJDBCException: Cannot open connection
CAUSED BY: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-jetty-########] Timeout: Pool empty. Unable to fetch a connection in 20 seconds, none available[size:75; busy:75; idle:0; lastwait:20000].
Caused by:
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
...

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log repeatedly show messages similar to:

2025-02-02 07:50:43,658 | DEBUG    | pool-jetty-800            | CSecurityManager               | Security context created for user <user>@<domain> with roles [Organization Administrator] | requestId=########-####-####-####-############,request=POST https://<vcd_url>/cloudapi/1.0.0/sessions,requestTime=1738482643440,remoteAddress=##.##.##.##:62484,userAgent=<none>,accept=application/json; version 37.0
2025-02-02 07:50:43,749 | ERROR    | auto-pool-eventPublishing-2-thread-1 | NotificationsAmqpPublisher     | Error processing notification com.vmware.vcloud.common.notification.Notification@#######, will not be able to retry. |
com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange '' in vhost '', class-id=60, method-id=40)

Environment

Cloud Director 10.5.1

Cause

This is caused by an external application attempting to interact with Cloud Director, and failing repeatedly.  This leads to socket exhaustion, and Cloud Director may become unavailable.

Resolution

Determine what is misconfigured in the application or script which is logging into Cloud Director, or configure it to stop trying after a number of failures so that it does not cause stability issues with Cloud Director.