A myriad of DatabaseNotReadyException errors are streaming through the SMP logs and many functions do not work

book

Article ID: 170081

calendar_today

Updated On:

Products

Management Platform (Formerly known as Notification Server)

Issue/Introduction

A long list of DatabaseNotReadyException errors are streaming through the SMP logs and many functions do not work

No console access.

NSE files are not being processed but instead are being thrown away.

Scheduled tasks are failing.

There is a river of errors similar to the following in all of the log entries:

Failed to open database connection. Current user is OurDomain\altirissv. Error: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
   [Altiris.NS.Exceptions.AeXException @ Altiris.NS]
   at Altiris.NS.Utilities.DbUtils.OpenConnection(SqlConnection dbConn)
   at Altiris.NS.Utilities.DbUtils.CreateConnection(String server, String catalog, String user, String password)
   at Altiris.NS.Utilities.DbUtils.CreateConnection(String configKey)
   at Altiris.NS.ContextManagement.AdminDatabaseContext.OpenDbConnection()
   at Altiris.Database.DatabaseContext`1.Initialize(Boolean beginTransaction, IsolationLevel isolationLevel)

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
   [System.Data.SqlClient.SqlException @ .Net SqlClient Data Provider]
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Altiris.NS.Utilities.DbUtils.OpenConnection(SqlConnection dbConn)

SQL Exception details: code=-1

 

Cause

The error points out that the database cannot be accessed, but it does not tell why. The following are things that should be checked and verified.

  1. The Coresettings.config contains reference to the correct SQL server name and instance in the "DBServer" tag.
  2. The database server specified in Coresettings.config replied to a ping and the route to it could be verified.
  3. The SQL service for the instance that houses the Symantec_CMDB database is started.
  4. The Service account, and or the SQL user as the database access account is still active and has permissions.
  5. The SQL logs did not have any information in them indicating authentication failures, nor did the Windows "Application" and "System" event logs.
  6. Verify the TCP/IP port SQL is listening on (default is 1433) is open. This would require opening the SQL Configuration tool on the SQL server and testing connections to it.
  7. Verify that UDP port 1434 is open.

Environment

Symantec Management Platform 7.6, 8.x

Resolution

All of the previous items were checked with no success in resolving the problem.  So the SQL box was restarted.

In this case that is what fixed the problem. The SQL box was unable to listen or respond on the alternate/custom port "13177" (detected by reading a Wireshark trace). Restarting the SQL server corrected the problem.