Asset Management Solution failing to configure AssetControl.config during an upgrade to 8.5

Products

• IT Management Suite
• Asset Management Solution

Issue/Introduction

The customer is trying to upgrade from 8.1 RU7 to 8.5 and then to 8.5 RU1. However, the upgrade to 8.5 failed.

According to the Install logs, it failed while trying to upgrade Asset Management Solution.

1) In one point during the upgrade to 8.5, the configuration process started with:

Beginning to start the task to evaluate asset costs.
------------------------------------------------------------------------------------------------
Process: AeXSvc (3640), Thread ID: 233, Module: Altiris.AssetControl.dll
Priority: 4, Source: Altiris.AssetControl.Settings.AssetCostsEvaluation.AssetCostsEvaluation_Process

2) Then, it stayed in a loop in this step for 2 hours:
Configuring task 43 of 78: Configure Asset Management Solution
------------------------------------------------------------------------------------------------
Process: SymantecInstallationManager (8972), Thread ID: 20, Module: SymantecInstallationManager.exe
Priority: 4, Source: Symantec.Installation.Automation.Output.ReportInfo

3) Then it failed:

Entry 1:
Failed to execute sql.

Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
....[removed section to simplify details]

SQL Exception details: code=-2

The wait operation timed out

....[removed section to simplify details]
------------------------------------------------------------------------------------------------
Process: AeXSvc (3640), Thread ID: 214, Module: Altiris.CMDB.Common.dll
Priority: 1, Source: Altiris.CMDB.Common.Utilites.SqlHelpers.ExecuteNonQueryEx

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Entry 2:
An error occurred during Asset Costs Evaluation Process.

Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
....[removed section to simplify details]

SQL Exception details: code=-2

The wait operation timed out

....[removed section to simplify details]

------------------------------------------------------------------------------------------------
Process: AeXSvc (3640), Thread ID: 214, Module: Altiris.AssetControl.dll
Priority: 1, Source: Altiris.AssetControl.Settings.AssetCostsEvaluation.AssetCostsEvaluation_Process

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Entry 3:
Failed to install the product from the file: D:\Program Files\Altiris\Asset\Config\AssetControl.config

Database context does not have open connection.
....[removed section to simplify details]

------------------------------------------------------------------------------------------------
Process: AeXSvc (3640), Thread ID: 214, Module: Altiris.NS.dll
Priority: 1, Source: Altiris.NS.Installation.ProductConfigurationWorker.ConfigureProductInternal
 

After this failure, multiple Solutions appears as installed but not configured. If you try to repair them, still fails under Asset Management Solution.

Entry 1:
Failed to execute sql.

Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
[System.Data.SqlClient.SqlException @ .Net SqlClient Data Provider]
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

....[removed section to simplify details]

SQL Exception details: code=-2

The wait operation timed out
[System.ComponentModel.Win32Exception @ ]

Exception logged from:
at Altiris.CMDB.Common.Utilites.SqlHelpers.ExecuteNonQueryEx(String, Boolean, System.Data.CommandType, Object[])
at Altiris.AssetControl.Settings.AssetCostsEvaluation.AssetCostsEvaluation_Process()
at Altiris.AssetControl.AssetControlInstallation.UpgradeAccountingPeriod()
at Altiris.AssetControl.AssetControlInstallation.OnInstallProduct(System.Xml.XmlNode)
at Altiris.NS.StandardItems.Product.ProductInstallation.InstallProduct()
at Altiris.AssetContractCommon.AssetContractCommonBaseInstallation.InstallProduct()

....[removed section to simplify details]

------------------------------------------------------------------------------------------------
Date: 2/9/2019 12:26:00 AM, Tick Count: 15563609 (04:19:23.6090000),
Process: AeXSvc (3640), Thread ID: 214, Module: Altiris.CMDB.Common.dll
Priority: 1, Source: Altiris.CMDB.Common.Utilites.SqlHelpers.ExecuteNonQueryEx
 

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Entry 2:
An error occurred during Asset Costs Evaluation Process.

Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
[System.Data.SqlClient.SqlException @ .Net SqlClient Data Provider]
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)

....[removed section to simplify details]

SQL Exception details: code=-2

The wait operation timed out
[System.ComponentModel.Win32Exception @ ]

Exception logged from:
at Altiris.AssetControl.Settings.AssetCostsEvaluation.AssetCostsEvaluation_Process()
at Altiris.AssetControl.AssetControlInstallation.UpgradeAccountingPeriod()
at Altiris.AssetControl.AssetControlInstallation.OnInstallProduct(System.Xml.XmlNode)
at Altiris.NS.StandardItems.Product.ProductInstallation.InstallProduct()
at Altiris.AssetContractCommon.AssetContractCommonBaseInstallation.InstallProduct()

....[removed section to simplify details]

------------------------------------------------------------------------------------------------
Date: 2/9/2019 12:26:00 AM, Tick Count: 15563609 (04:19:23.6090000),
Process: AeXSvc (3640), Thread ID: 214, Module: Altiris.AssetControl.dll
Priority: 1, Source: Altiris.AssetControl.Settings.AssetCostsEvaluation.AssetCostsEvaluation_Process
 

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Entry 3:
Failed to install the product from the file: D:\Program Files\Altiris\Asset\Config\AssetControl.config

Database context does not have open connection.
[Altiris.Database.DeadDatabaseContextException @ Altiris.Database]

....[removed section to simplify details]

Exception logged from:
at Altiris.NS.Installation.ProductConfigurationWorker.ConfigureProductInternal(String, Boolean, Altiris.NS.Serialization.SerializationMode)

....[removed section to simplify details]

------------------------------------------------------------------------------------------------
Date: 2/9/2019 12:26:00 AM, Tick Count: 15563609 (04:19:23.6090000),
Process: AeXSvc (3640), Thread ID: 214, Module: Altiris.NS.dll
Priority: 1, Source: Altiris.NS.Installation.ProductConfigurationWorker.ConfigureProductInternal
 

Environment

Upgrade from ITMS 8.1 RU7 to 8.5

Cause

Known issue. The problem resides on a timeout caused by data been inserted to a specific table during the upgrade:

INSERT INTO AssetAssociationChangingHistory

This process was too expensive and required the creation of 2 indexes to help with it.

Resolution

This issue has been reported to the Symantec Development team.

After grabbing the affected query and ran an Estimated SQL Execution Plan, we identified that for whatever reason SQL Server was asking for 2 indexes. We created them
using what SQL Execution plan itself recommended:

--Index 1
USE [Symantec_CMDB]
GO
CREATE NONCLUSTERED INDEX [IDX_ResourceInventoryHistory_Support] --custom index name added
ON [dbo].[ResourceInventoryHistory] ([InvClassGuid])
INCLUDE ([SnapshotId],[InventoryDate])
GO

--Index 2
USE [Symantec_CMDB]
GO
CREATE NONCLUSTERED INDEX [IDX_ResourceInventoryHistory_Support2] --custom index name added
ON [dbo].[ResourceInventoryHistory] ([InvClassGuid])
INCLUDE ([ResourceGuid],[InventoryDate])
GO

After that, SIM was able to pass Asset Management Solution configuration portion and finish.