Various tasks fail while calling stored procedures with errors: "Error executing query usp_"
search cancel

Various tasks fail while calling stored procedures with errors: "Error executing query usp_"

book

Article ID: 329070

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:

  • After upgrading or migrating to vRealize Automation (formerly known as vCloud Automation Center) 6.1, 6.2.x or later, various tasks (such as data collections, provisioning, destroying machines and other tasks) fail.
  • After a new installation of vRealize Automation 6.1 or later, you see tasks failing due to stored procedure failures.
  • In the vRealize Automation admin log files, you may see Microsoft Distributed Transaction Coordinator (MSDTC) errors.
  • You see errors similar to these within Infrastructure > Monitoring > Logs:

    Error processing workflow creation Error executing query usp_SearchInitializingRequestVirtualMachines Inner Exception: Error executing query usp_SelectGroup
    DataBaseStatsService: ignoring exception: Error executing query usp_SelectAgent Inner Exception: Error executing query usp_SelectAgentCapabilities
    stack trace at DynamicOps.VMPS.DataAccess.ConnectionFactory`1.ExecuteReader(RecordReader`1 recordReader, String storedProcedure, String whereClause, Boolean firstRecordOnly, SqlParameter[] collection) at DynamicOps.VMPS.DataAccess.DataAccessLayer.SearchInitializingRequestVirtualMachines(Int32 resultCount) at DynamicOps.VMPS.DataAccess.DataAccessLayer.<>c__DisplayClass181.

  • In your IaaS Server Event Logs, you see the error similar to:
    A caller has attempted to propagate a transaction to a remote system, but MSDTC network DTC access is currently disabled on machine 'SQL_MACHINE'."



Environment

VMware vRealize Automation 6.x.x
VMware vRealize Automation 7.x

Cause

This issue occurs due to an MSDTC failure at the SQL server side as MSDTC is either not configured or stopped on the SQL server or cluster. Another possible cause for this is if your IaaS servers and SQL database server are cloned and their Windows security identifier has not been changed (CID) through sysprep or other method.

Resolution

To resolve this issue, execute the following procedure:
  1. Log in to the SQL server or cluster primary node.
  2. Stop the Manager Service.
  3. Stop the SQL Server service.
  4. Open a command prompt on the machine with the Manager Service and issue the following command:

    msdtc -uninstall
  5. Open a registry editor on the Manager Service and delete the following keys if they exist:

    HKLM/Software/Microsoft/Software/MSDTC
    HKLM/System/CurrentControlSet/Services/MSDTC
    HKEY_CLASSES_ROOTCID
  6. Reboot the machine with the Manager Service.
  7. Open a command prompt on the machine with the Manager Service and issue the following command:

    msdtc -install
  8. Perform steps 3-6 on the machine running the SQL Server.

    This procedure generates new CID values for MSDTC on both servers.
Notes:
  • For clustered MSDTC instances make sure the settings are enabled on both the Local DTC and the clustered DTC on all hosts in the cluster.
  • It is important to note that the Microsoft Distributed Transaction Coordinator (MSDTC) is uninstalled and reinstalled if the SQL Server and vRealize Automation virtual machines are cloned from the same base image without running sysprep before cloning (not a best practice). The reason for this is that the original and cloned virtual machine have the same CID values that causes communication failures between the vRealize Automation and the SQL server.
  • Also, after reinstalling MSDTC, you need to set the Distributed Transaction Coordinator service to Automatic (Delayed Start) and turn it on.
  • Even though the Microsoft Distributed Transaction Coordinator is enabled, the distributed transaction may fail if the firewall is turned on.
  • The Manager Service machine hosting the Manager Service should be able to resolve the NETBIOS name of the IaaS MS-SQL DB machine.
    If it is not achievable due to any reason, the workaround will be to add the NETBIOS name of the DB machine to the /etc/hosts file of the Manager Service machine and restarting Manager Service.

Additional Information

Impact/Risks:
Directly modifying the Windows registry can harm your computer. Take a backup and do at your own risk.