SCM Performance Troubleshooting

Products

• CA Harvest Software Change Manager - OpenMake Meister
• CA Harvest Software Change Manager

Issue/Introduction

If you, as the Harvest Administrator, receive a call or email from an end user that SCM performance is slow, then this document attempts to provide some steps to help you identify the problem area and the root cause of the problem.

Environment

CA Harvest Software Change Manager all versions and platforms

Resolution

There are many variables to consider in relation to SCM performance. Since a database, network, server, and client application all must work in concert in order to process SCM transactions such as check out / check in in a timely fashion, each facet must be analyzed so that possible sources of the problem can be ruled in or out. Once an area has been identified that is causing the disruption then appropriate analysis needs to be further applied in order to narrow down the root cause. For example, if analysis reveals that the problem appears to stem from the client machine so then immediate focus can be on the client environment settings to try to determine if anything recently has changed such as a possible corporate enterprise software change has been recently pushed to the client or it may be possible that the user is running a concurrent application that is using high CPU cycles on the machine. These possibilities have been exposed in other customer issues as the root cause, so if performance has been nominal and now it is performing poorly then most likely something recently within the environment has changed to affect this condition and this should be taken into account with your analysis.

When an end user reports an SCM performance problem, you must gather as much information about the incident as possible:

1. What SCM client application was being used? Workbench? Visual Studio? Eclipse? Command line utility such as hco?  Harweb?
   
   
2. What is the client SCM version?
   
   
3. Did this work ok before? if so, what has changed recently?
   
   
4. Is the problem persistent or intermittent?
   
   
5. Is this problem affecting all the users or just one or a few?
   
   
6. What type of activity was the end user performing when the problem happened?
   
   
7. Did the user encounter any error message? If so please provide a screen shot of that message.
   
   
8. Use hdbgctrl command line utility on the broker machine to monitor and log the client's processing.

Network:

1. Can the user ping the Broker (or Agent if Agent problem) from his client machine? Can Broker (or Agent) machine ping the user's machine?
   
   
2. Is a direct connect port range implemented?
   
   
3. If Telnet is available, what is the result of the "telnet <broker> <port>" command from the user's machine?  As an alternative on a Windows computer, the PowerShell Test-NetConnection commandlet can be used.
   
   
4. Are the proper ports opened on the Firewall?
   
   
5. Can the broker ping the database machine (if remote from the broker) and vice versa?

Server:

1. Is broker (bkrd) service running and the minimum number of hservers and rtserver running?
   
   
2. Did restarting the broker clear up the problem?
   
   
3. Do all of the hserver processes connect to the database successfully?
   
   
4. Does increasing the number of minimum hservers correct the problem?
   
   
5. If using a direct connect port range, does increasing the number of ports and hservers have a positive affect?

Database:

1. If your database is on a remote machine, has the database been restarted and if so, did this clear up the problem?
   
   
2. Did the database generate any problem alerts?
   
   
3. In many cases, rebuilding the Harvest table indexes and locking Harvest temp table statistics (for an Oracle database) have provided markedly performance improvements.

If all servers and database systems appear to be online and the network and firewall configurations appear to be nominal then the problem of performance may be SQL related.

The Harvest broker has the capability to generate a set of HBroker and HServer log files that can record all SCM transactions down to the SQL statements that are applied to the database and the returned results. The amount of detail about SCM transactions to be recorded in the log files can be adjusted to meet your needs. This can be done in either of two different ways. The old method involves configuring the Broker's HServer.arg file to add the -logging=n (n=1 to 4, generally level 4 is used to provide SQL statement logging) line and then restarting the broker for the log level to take affect. This manual effort to generate logs can be accomplished at all Harvest release versions. The downside is that the Broker must be restarted and this could adversely affect production. Beginning with R12.1 SP1 and above, a new command line utility has been provided called "hdbgctrl" that can be used to adjust the logging level and generate the HBroker and HServer logs at the appropriate log level without requiring a restart of the Broker. Bear in mind that the use of hdbgctrl will override any current log level settings set in the HServer.arg file. 

Details on usage and examples of the hdbgctrl utility can be found here: hdbgctrl Command - Debug Control

Keep in mind that the Harvest HBroker to HServer logs are a one-to-many relationship. One HBroker logs can reference one or more HServer logs. This is because in Harvest, the Broker manages a pool of HServers. The Broker initially receives the client SCM transaction request and then checks the HServer pool to see if any HServer processes are free and then passes the transaction to the HServer for it to process with the database. The HServer then returns the transaction results directly to the client machine and then reports to the Broker that it is free for the next transaction.  When sending log files to Broadcom support for analysis, it is important to provide a complete set of log files, the HBroker log and all supporting HServer logs.  The log files are found in the %CA_SCM_HOME%\log folder and follow the name format [date]HBroker[pid].log and [date]HServer[pid].log.

Additional Information

For more information about where log files for the various components of Harvest can be found, refer to this article: Location of the Harvest log files