Over the past 2 weeks, the performance of a CA XCOM standard file transfer of 13GB (non-compressed) has suddenly started to degrade significantly and is taking approximately 4 times longer to complete i.e. 2-3 hours instead of 30-40 minutes. There are no errors or change in the environment or command that indicates a possible root cause.
However, in the $XCOM_HOME/Q directory it has been noticed that there is a build-up of .MBR files from the date when the XCOM performance started to slow down.
Release : 11.6
Component : CA XCOM Data Transport for AIX
The $XCOM_HOME/Q directory contains the .IDX file and .MBR files, each .MBR corresponding to a transfer on that system.
If for whatever reason there are old .MBR files that do not show when command "xcomqm -La" is issued, then those are .MBR files from old transfers which were not properly removed from the queue due to a specific reason. Finding the reason is impossible, but if xcomd is abnormally terminated then old .MBR files can be the result because XCOM could not perform the necessary tasks of waiting for the transfer to end or cleanup. In other words, "the rug was pulled out from under XCOM".
The .MBR files can increase in number over time and it will affect how many transfers can be in the queue at any given time because of the value of MAX_QUEUE_ENTRIES in the xcom.glb.
In summary, it is best practice to always stop xcomd normally with the documented command of "xcomd -s" and also once in a while review what is in the Q directory. If there are old or corrupted .MBR files then stop xcomd and remove the old .MBR files.
In this case stopping xcomd, removing the .MBR files and restarting xcomd restored the performance to normal.