Detailed CORA Trouble-Shooting Guide
Article ID: 53063
Updated On:
Products
Issue/Introduction
CA Support recommend that the following steps be reviewed to diagnose CORA issues.
CORA underpins all integration between different CA products and all data loading via product utilities into CA products.
If CORA is not working, information may be missing, duplicated or otherwise in error.
Environment
CA Service Management 11.x and higher (includes CA Service Desk and IT Asset Manager)
Resolution
CORA Integration with Service Desk and CMDB
Overview and Definition:
Note: This information is supplementary to the official product documentation. Where a conflict arises, the product documentation takes precedence.
The key to all integration between CA products is "CORA."
The CA Service Management product uses the common MDB schema to store and manage their data. The Common Registration API (CORA) is the interface through which these configuration items are registered. It is the only source for updating these tables, ensuring that configuration item data flows consistently, thereby supporting the data and referential integrity of the MDB's master configuration item data model.
CORA's purpose is to prevent the same object from being placed in the MDB multiple times by the many different methods of discovery in use between products that access the MDB. It stands between products and any update to a CA product goes through CORA rules.
It is important to note that CORA rules are not invoked by means that are our outside of the product. For example, a direct database load using native database utilities would not go through CORA. However, a database load using any CA product utilities, such as pdm_load for Service Desk, does go through CORA.
The logic of CORA is to identify each configuration item and map the same configuration item to the products which register with the same attributes.
See the "The MDB, Hardware Configuration items and CORA" for the essential information to understand how CORA works. This document is the foundation of understanding CORA and associated problems. Filename CORA_MDB_and_Assets_SC.pdf
What is a Configuration Item?
While there is a strict definition of a "Configuration Item" within the ITIL methodology, one which is followed closely be many CA products, it is important to note that in common usage the terms Assets and Configuration Items are often used interchangeably. It may be important to understand the context in which these terms are used in order to understand an issue. Here are some of the ways that the terms may be used differently:
Service Desk/CMDB - The role of an configuration item is any resource that provides value to the enterprise, which may not be an IT resource. For example, a server, a Document or a Role.
In most cases an "Asset" or "Configuration Item" in the Service Desk/CMDB world refers to an IT resource, usually a piece of hardware or software.
For IT Asset Manager or Asset Portfolio Manager (APM), it may also be a software license, a catalogue of available laptops, desktops for new hires and so on.
For legacy DSM and NSM components, users may define Assets as hardware items to discovered, mapped and monitored.
The CORA API provides a way to link discovered/monitored assets with the same entity on the other side where it is the owned resource for UAPM, CMDB and Service Desk.
See also pdm_reg_assets, pdm_discimp and pdm_nsmimp.
CA Support Troubleshooting Guidelines
If an integration between CA products appears to satisfy other setup requirements, and yet unanticipated results are seen, such as duplicate or missing items, it could be an indication that a review of CORA is required.
The underlying integration method should be reviewed as a preliminary step though. For example, an unchanged integration setup which has been confirmed to have been working in the past which now produces unexpected duplicates could indicate that time may be best spent on analysing around CORA. An integration which has never worked, or which fails on single item test cases may have analysis time better spent on the integration setup within the products.
The Quick Fix Attempt
The following procedure is a common resolution path to many CORA issues.
In particular, it should be followed for all "duplicate" configuration item problems.
- Apply the latest CORA Cumulative Patch.
Where duplication has occurred, apply this fix to all:
- DSM servers
- UAPM servers
- Service desk servers
- NSM servers - any management machine with CCS
Use APPLYPTF to apply the CORA patch on all relevant products by using the appropriate JCL file for the product. If there are two products installed on the same machine, apply the patch twice selecting the product related JCL File.
Note: Crash dumps for CORA that require CA Sustaining Engineering review must be taken from CORA Cumulative Patch 3 or later as the viewing facility is only included from this version. If on an older version, move to the latest version and regenerate the output. - DSM servers
- Run the CORA Cleanup tool to re-register all assets with CORA.
This procedure addresses so many of the common technical issues that it is often worth attempting this resolution path before going further.
Information Gathering and Trouble-Shooting Guide
- Define the problem as clearly as possible.
This helps to set the scope of the problem.
It should help to direct investigation into the most likely areas.
It also provides a benchmark to identify when the issue is resolved. - Search support.broadcom.com for known problems and solutions. Search against the products involved and against the CORA product.
If there is a known problem, apply the solution and typically also run the CORA Cleanup to re-register all assets in the MDB.
Known Problems:
Known problems with DSM & NSM
Unmanaged Assets (Including Routers, Hubs, & Switches) are assets discovered by NSM (CCS) and registered with CORA and the same Asset NOT registered by DSM in CORA
Unmanaged Assets not displayed. Ensure that the Assets are discovered by CCS and can be seen in the 2-D map. Apply the test fix T44D277 and then run the CORA Cleanup tool provided in the test fix to re-register assets in CORA
Known problems with DSM & UAPM
UAPM Reconciliation Job not working - UAPM Reconciliation Job finds matches based on the 'Criteria' provided in the Job (like Serial Number, Asset/Resource Tag, or Host Name)
The job detects the asset_source_uuid values of the records from UAPM & DSM and passes them to CORA to 'Move' (LINK) them to same LOGICAL Asset.
Problem seen so far
Asset_source_uuid passed to CORA do not Exist in CORA - CORA Cleanup usually resolves this problems.
Logical_asset_uuid is NULL for DSM Assets in ca_asset_source table - Sometimes DSM might not be able to register Asset with CORA and thus adds a record in ca_asset_source table with the asset_source_uuid value and NULL for Logical_asset_uuid as the record is necessary to store Software Signatures information in DSM. CORA Cleanup resolves this problem as well and the test fix T44D278 might be needed depending on the DSMVER of the client.
Known problems with DSM & Service Desk
Discovered Asset List Not showing the Assets - CORA cleanup should resolve this problem
Discovered Asset List showing Duplicate Assets - CORA Cleanup will resolve this problem as well. Another point to note is if there are multiple DNS/MAC Address pairs registered by DSM for an Asset, ALL will be displayed thus the perception that the Duplicate records are being displayed
Please check if ALL the attributes of the records being shown as Duplicates are indeed same and nothing is different
Discovered Asset List showing Junk Characters in DNS Names - DSM Linux Assets reports junk characters as part of DNS. Test Fix T5B1039 (TNGAMO-2819) resolves the problem and is also part of DSM 11.2 C2 published patch QO96925.
Import Assets Failed in Service Desk - CORA Cleanup will resolve this problem.
Upgrading CORA - The CORA deliverables are mostly dll files that are loaded by other processes. If you are trying to upgrade CORA, all the processes that load the dlls need to be turned off. If you still have problems you can use "Process Explorer" to locate the process that is still running that has the dlls locked. If you still can't find the process then check the file permissions and make sure that no "Deny" access is set for any user. - Define the environment
List all applications and their versions in the environment that use the MDB and Assets.
List operating systems and names of the involved servers.
As there is a common MDB, other processes may be impacting the asset tables apart from the expected ones.
Knowing the environment helps to rule in or out various problems or solutions.
Check if either Service Desk and CMDB are installed separately, or are both in use? - Collect logs and screenshots of the problem.
Collect the CORACLEANUP.LOG file, created in the same folder that the CORA Cleanup is executed,
Collect screen shots of the problem as well.
Review the logs for error messages that occur at the time of the problem and check against support.broadcom.com. - Confirm the expected behaviour of CORA
Review the document 'MDB, Hardware Assets and CORA' in order to find out if CORA is behaving as per the documented design or not.
CORA_MDB_and_Assets_SC.pdf.
The majority of 'procedural' CORA problems are resolved by an understanding of how CORA works and the algorithms that it is applying. Questions such as 'I have supplied fields X, Y and Z' but no records were imported.' In particular, the CORA uniqueness and test for duplicate matrices are invaluable. - Reproduce the issue on another system.
Extract the data from the non-working system and load onto a clean test system.
This may then be analysed, and solutions tried on the test system without impacting the production system.
It is a good idea to take a backup of the test system both before loading the above, and a separate backup after the load has been successfully done. You may then revert to either state to test resolutions.
The following describes the procedure for a SQL database and assumes that the Service Desk product is involved. It would need to be altered to work on another database type such as Oracle by the database administrator.
The reason why native database commands are used and not CA product commands such as the Service Desk commands pdm_extract/pdm_backup, is that they are not reliable when troubleshooting CORA problems. The reference columns are described in the product files, such as ddict.sch file for Service Desk. Columns that are not referenced in the Service Desk ddict.sch would not be found.
Using native database utilities works around this design.
Make the test server Service Desk Setup have the same Schema Level as that of the problem system.
Since we are trying to troubleshoot CORA problem, we are mainly worried about usp_owned_resource table for any added columns in R11.x, The ca_owned_resource is for any customized columns that got migrated from Service Desk 6.0 or earlier..
On the problem system's database server, gather the table information via attached batch file to collect all the tables relevant to CORA: Exp_Imp_Tables.zip. The batch file uses BCP (Bulk Copy feature of SQL Server) to export the tables, and is included in the file CORA_Troubleshooting Utilities.zip.
Run the Export_Tables.bat on the machine where SQL Client/Server is installed through the command line:
Export_Tables <SQL SERVER NAME> <SQL USER> <SQL PASSWORD>
Replace <SQL SERVER NAME> with the SQL server name of MDB
<SQL USER> as sa/SQL Admin user
<SQL PASSWORD> as the password of the SQL user
Once the batch file completes, collect the files generated.
Check the .OUT files to ensure that there were no errors during the process.
Move the data to a test machine and load the data with the Import_Tables.bat.
If you come across any problems, you may need to drop the constraint on that particular table.
Analyse the problem on the new test server. - Get Expert Assistance
If the problem is unresolved at this point, advanced trouble-shooting may be required to analyse debug log files or simply as a second opinion. Even sites with access to advanced skills who wish to continue the debugging process themselves from this point, may find it worthwhile logging an issue with CA Support so that the issue history is established.
Be prepared to provide the information gathered in the steps to date.
Here is the recommended checklist of information to gather when logging complex CORA issues:
Issue summary:
Problem description summary.
Environment:
For an issue where no known solution exist document why you think that this is a CORA issue and what the expected outcome is.
Document the steps from this Trouble Shooting Guide that were done. Use best judgement to decide which ones are appropriate - The Quick Fix Attempt steps are recommended in most cases.
If the issue is reproducible provide the steps.
Did testing with the CORA Test Tool, which by-passes the product, identify that it was a product problem, or did the same problem occur?
Business Impact:
How urgent is the issue? What are the expectations?
If logging as a Severity 1 issue, document with the 24*7 client contact number.
Reproduction Machine details:
Note if there is a machine or VMWare image where the problem can be reproduced, or if it is possible to do a remote connection via CA SupportBridge Automation Software.
Files collected and attached to the issue:
Document all files attached to the issue and what each file contains, unless obvious.
Contacts:
List alternate client contacts if required. - Test for the problem outside of CORA.
The CORA Asset Register Tool is used to register Assets directly with CORA to check and reproduce the problem. That is, it interfaces directly to CORA and does not go through other CA products such as Service Desk, DSM, NSM etc.
The utility is CoraTestTool.exe and is included in the CORA_Troubleshooting Utilities.zip.
Note: This utility is a CA Support utility and not an officially released CA product. It is provided to assist with trouble-shooting problems 'as is' and no support will be provided for this product or for unintended consequences of its use. No further documentation other than comes with the utility will be provided. - Review the CORAOUT.LOG file in more depth.
The utility 'CORA Log Analyser' may be used to analyse the CORAOUT.LOG file further and is included in the CORA_Troubleshooting Utilities.zip.
Note: This utility is a CA Support utility and not an officially released CA product. It is provided to assist with trouble-shooting problems 'as is' and no support will be provided for this product or for unintended consequences of its use. No further documentation other than comes with the utility will be provided. - Setting CORA to Debug mode.
If the normal logging levels are insufficient to reveal problems, the logging level may be increased.
Set the CORA in Debug mode and collect CORA logs with the following steps.
Go to the folder in the environment path variable %GC_CONFIG%
Open the file CoraLog4cplus.cfg
Set log4cplus.logger.Cora=DEBUG
log4cplus.appender.CoraAppender1.MaxFileSize=50MB
log4cplus.appender.CoraAppender1.MaxBackupIndex=5
Also, perform a search for CoraLog4cplus.cfg in the folder of the calling application and set the DEBUG value
Coraout*.log file(s) will be created in the path specified in entry log4cplus.appender.CoraAppender1.File
If no path provided, then the Coraout.log file will be created in the folder of the calling application/CORA_LOGS folder in %GC_CONFIG% folder.
Re-start all the product Services/Plugin(s) for the changes to take affect
- CAF STOP
- UAPM Services & IIS
- NSM & CCS Discovery Services
- Service Desk Services
Also, check the Asset(s) for which the problem exists along with the information as Host Name, Serial Number, Asset Tag, DNS, MAC Address, and Label to check for the problem in the logs
The following query can be run in the SQL Query Analyser by selecting the 'MDB' to check if the information available in CORA is correct
select asset_source_uuid, ca_logical_asset.logical_asset_uuid, ca_asset.asset_uuid, host_name, serial_number, label, asset_tag, dns_name, mac_address
from ca_asset_source, ca_logical_asset, ca_asset, ca_logical_asset_property
where host_name like '<HOST NAME>'
and ca_asset.asset_uuid = ca_logical_asset.asset_uuid
and ca_logical_asset.logical_asset_uuid = ca_asset_source.logical_asset_uuid
and ca_logical_asset.logical_asset_uuid = ca_logical_asset_property.logical_asset_uuid
The above query lists all the attributes of the Asset as per the WHERE clause
The WHERE Clause can be changed to check on the other properties like Serial_Number, Asset_Tag, DNS_NAME, MAC_ADDRESS, LABEL as well - CAF STOP
- Diagnostic Report for CORA issues.
The purpose of this test utility is undocumented.
The file cora_MDB_diag.rpt is included in the CORA_Troubleshooting Utilities.zip.
Usage: rpt_srv.exe mdb.rpt > diag_rpt.txt
Note: This utility is a CA Support utility and not an officially released CA product. It is provided to assist with trouble-shooting problems 'as is' and no support will be provided for this product or for unintended consequences of its use. No further documentation other than comes with the utility will be provided.
For information on 'pdm_reg_assets' and the r11 migration procedure, contact CA Support.
Additional Information
CORA_Troubleshooting Utilities.zip is included in this tech article. It is attached as TEC484950.zip.
Attachments
Feedback
