General ITCM Troubleshooting Steps

Products

• CA Client Automation - IT Client Manager
• CA Client Automation

Issue/Introduction

This document describes some general troubleshooting steps of ITCM.

Environment

Client Automation - All versions

Resolution

Recycling ITCM services:
Often times, recycling the services associated with ITCM can resolve a general amount of issues.

Here are the steps to fully stop all ITCM-related product services:
caf stop
caf stop ext
cam change disabled
camclose
csampmux stop

Alternate commands:
If "caf stop" gives an error, e.g. "failed to connect to local CAF service", then use: "caf kill all" instead.
If "camclose" gives a timeout error, then use: "taskkill /im cam.exe /f"
If "csampmux stop" gives an error, then use: "taskkill /im csampmux.exe /f"

Once services are stopped, delete the notification server cache file:
..\CA\DSM\appdata\cfnotsrvd.dat

To restart services:
csampmux start
cam change auto & cam start -c -l
caf start

If the system has the .NET framework installed, you could also use WinOffline to recycle services.
The latest WinOffline at the time of publication is available here:

CA Client Automation > Latest WinOffline -- 2019.01.10

Connectivity troubleshooting:
For connectivity type issues, ITCM uses the following general communication ports:
UDP 4104 -- CAM Messaging
TCP 4728 -- CSAM Bulk traffic (e.g. file uploads/downloads)

However, it's common place for CAM messaging to be configured to use TCP, in which case there are additional port requirements:
UDP 4104 -- CAM UDP Messaging
TCP 4728 -- CSAM Bulk traffic (e.g. file uploads/downloads)
TCP 7163 -- CSAM traffic for CAM TCP

Note: These ports must be opened bi-directionally!  That is, either side is able to initiate traffic to the other.

If you are using the infrastructure deployment wizard to install new agents, you need the following ports opened, in addition to the normal communication ports:
UDP 4104 -- CAM UDP Messaging
TCP 4728 -- CSAM Bulk traffic (e.g. file uploads/downloads)
TCP 7163 -- CSAM traffic for CAM TCP
TCP 7 -- Echo Request
UDP/TCP 135-139, TCP 445 -- RPC/SMB Share ports

Note: For Unix/Linux deployments, you also need the SSH/telnet port opened, typically TCP 22/23 respectively.

To test CAM connectivity, you can use the "camping" utility:
camping <hostname or IP>

This will send a 64-byte test message from CAM, to the specified host or IP.  If CAM is running/reachable there, it will receive the message and send a reply message.

In a growing number of environments, UDP traffic over a certain size is becoming filtered.  You can test sending various size camping's using a series of commands to find if your environment may have such filtering:
camping <hostname or IP> -s 128
camping <hostname or IP> -s 256
camping <hostname or IP> -s 512
camping <hostname or IP> -s 1024
camping <hostname or IP> -s 2048
camping <hostname or IP> -s 4096
camping <hostname or IP> -s 8192
camping <hostname or IP> -s 16384
camping <hostname or IP> -s 32768

The tests DO NOT have to be even powers of 2.  In some cases, network latency and RTT may prevent a  reply from being received within the 1 second wait time that CAM expects.  To see additional camping options, run camping --help.

If your environment is camping the UDP fragment size, you can add this line to your *CONFIG section in your %cai_msq%\cam.cfg file:
fragment_size = xxx
Where xxx is a fragment size that is comfortably below the maximum you found in your environment by running camping tests of varying packet sizes.

If you don't have a cam.cfg file in your environment, first generate a save.cfg file with:
camsave config

This will create a %cai_msq%\save.cfg file.  Edit it to remove all paths from the *PATHS section, as this will be a dump of running connections from memory, and it is not desirable to have CAM reload these connections every time it starts up.  Once the fragment_size parameter is added to the *CONFIG section, save as cam.cfg and recycle services again.

To test CAF connectivity, which will test both the CAF (application layer) and CAM (messaging layer) connectivity:
caf ping <hostname or IP>

If CAMPING is working, but CAF PING is not working, it can indicate a difference in encryption.  For example, one endpoint is using the out of the box ITCM certificates, while the other is using some custom ITCM certificates.  It could also mean the clocks are different by more than 10 minutes between the two endpoints.

Log Files:
If opening a support case, it's always recommended to attach the DSM/ITCM logs from the affected systems.

The ITCM logs can be found here:
..\CA\DSM\logs

The default location is:
C:\Program Files (x86)\CA\DSM\logs

Agent Registration:
To check which scalability server (or domain manager) and agent is registered with:
caf setserveraddress

Note: Every domain manager has a built-in scalability server.  Enterprise managers do not have scalability servers, hence agents cannot register directly with an enterprise manager.

To change the agent's server address:
caf setserveraddress <FQDN, hostname, or IP of new SS>

For example:
caf setserveraddress caprica14.example.com

To force an agent to re-register:
caf register all

By default, an agent will re-register daily.  By running "caf register all", it will prompt the agent to run a registration with the scalability server.  Registration is also event-based, and will happen if the system is rebooted, a user logs on, or if there is a change in network address.