Portal services fail to maintain open connections to the remote MySql host.

Products

• Network Observability
• CA Performance Management

Issue/Introduction

Portal services fail to maintain open connections to the remote MySql host.

 

Customer runs a remote MySql install with it on a different host from the Portal web server services.

 

When there are firewalls in the customer environment between the Portal webserver host and the MySql DB server host, it will and can often lead to connectivty problems.

 

As a result of the broken and inconsistent connections failures and performance issues are seen in Portal requiring regular service restarts to reset/resolve the issue.

 

These are the errors we've observed that expose the problem. From the DMService.log:

 

WARN  | RIB-Query-38             | 2025-01-13 11:43:30,259 | org.springframework.jdbc.support.SQLErrorCodesFactory            
     | Error while extracting database name
org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: Connection was closed in SingleConnectionDataSource. Check that user code checks shouldClose() before closing Connections, or set 'suppressClose' to 'true'
...
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: Connection was closed in SingleConnectionDataSource. Check that user code checks shouldClose() before closing Connections, or set 'suppressClose' to 'true'
...
Caused by: java.sql.SQLException: Connection was closed in SingleConnectionDataSource. Check that user code checks shouldClose() before closing Connections, or set 'suppressClose' to 'true'

Environment

All supported DX NetOps Performance Management releases

Cause

Due to network connectivity issues, often caused by firewall timeout rules or other environmental impediments.

Resolution

Resolve the network and environmental issues impacting the connection consistency.

Broadcom will also endeavor to improve the Portal services MySql connections, ensuring they handle exceptions thrown on already closed connections, retrying the failed command after generating/getting a new connection. This is being tracked through defect ID DE165071 which is yet to have an ETA for a fix release set. This article will be updated when a release ETA is made available from engineering. This is current as of January 2025.