Search Server Troubleshooting

Products

CA Service Management - Service Desk Manager CA Service Desk Manager CA Service Catalog

Issue/Introduction

Search Server, or Elastic Search is a component that is leveraged by xFlow and Service Point to access content from Service Desk and Service Catalog. The following is a guide to how the Search Server component operates and what to look for in troubleshooting.

This article will be of value when troubleshooting scenarios such as missing Service Desk ticket content in Service Point as well as missing Catalog offering data in Service Point.

Environment

Release : 17.3 and higher

Component : SDM - xFlow

Resolution

For SM 17.3 GA to RU5, the ca_es_cluster.log file is located in C:\Program Files\CA\SC\CASearchServer\elasticsearch-2.1.1\logs

To set the logging into debug mode for this release (17.3 GA to 17.3 RU5), modify the line "es.logger.level: INFO" to "es.logger.level: DEBUG" in elastic search server install\elasticsearch-2.1.1\config\logging.yml file and recycle Search Service
From SM 17.3 RU6 onward, the ca_es_cluster.log file is located in C:\Program Files\CA\SC\CASearchServer\elasticsearch-7.10.2\logs
For SM 17.4 builds, the ca_es_cluster.log file is located in C:\Program Files\CA\SC\CASearchServer\elasticsearch-7.17.6\logs

To activate debugging (17.3 RU6 and up, and 17.4 builds), locate file "log4j2.properties" in C:\Program Files\CA\SC\CASearchServer\elasticsearch-XXXX\config, where XXXX is the current version of ElasticSearch present, depending on the build of Service Management. Backup, then edit the file in a text editor and locate this line:
```
rootLogger.level = info
```
Change the line to read as:
```
rootLogger.level = DEBUG
```
Save the "log4j2.properties" file, then recycle Search Service

When performing an initial load (pdm_es_initial_load.bat) or a subsequent index rebuild (pdm_es_rebuild_index.bat), file initial_load.log will be of value. This log file can be located in the C:\Program Files\CA\SC\CASearchServer\search\logs directory.

Service Desk/Search Server Interaction

Within SDM, under pdm_status, there will be an entry for "es_ebl" or the "Search Deamon" process. For conventional configurations, this process is located on the Primary SDM Server. For Advance Availability, the process is on both the Background and the App server running xFlow. Note that there is a known cosmetic spelling error for "Deamon". The "es_ebl" process will connect to Search Server to pass on content from SDM.

How this works is when an entry in SDM is recorded that needs to be passed on to Search Server, the "es_ebl" process will write an entry to the backend database table "es_events", then a "Push" is done to write the content over to the appropriate index in Search Server. Once the push to Search Server is completed, the corresponding entry in the database table "es_events" is deleted. Normally, the "es_events" table is empty, unless there is data to be pushed over to ElasticSearch. Once it is pushed to SS, the "es_events" table entries is deleted.

Log file of interest is the esEvents.log in the SDM log directory. One should examine this log for troubleshooting concerns involving ticket data from SDM to be transferred across to Search Server. To turn up the logging, one must update the esLog4j.properties file in NX_ROOT\site\cfg to debug levels, then run "pdm_bounce es_ebl" to restart the SDM "Search Deamon" process.

Service Catalog/Search Server Interaction

Within Catalog, the interaction is simpler. Catalog will make a direct connection to the Search Server to push its contents over to the appropriate index stored in Search Server.

Log file of interest is the Catalog view.log. One should examine this log for troubleshooting concerns involving request data and offering data from Catalog to be transferred across to Search Server. There will be entries for the Search Server interactions if one searches for "ESClient". View.log would need to have debugging enabled to view the content.

xFlow and Service Point/Search Server Interaction

xFlow and Service Point will utilise the Search Microservice defined in xFlow to interface and obtain materials from Search Server. One may consider turning up logging on the Search Microservice to view the content and materials that xFlow and Service Point obtains for a given transaction between its Search Microservice and the Search Server. Logs of interest will be the searchMS_debug.log and its derivative logs in the xFlow install directory.

Direct Search Server examination

The most common method to view the contents of Search Server's indexes is to utilise the Chrome plugin "ElasticSearch Head". A common activity is to use this functionalty to directly examine a given index stored in Search Server to determine its contents and accuracy when SDM or Catalog writes to the indexes and xFlow or Service Point reads from the indexes.

Information on the plugin is available here:

Once the plugin has been installed on your Chrome browser instance, you can access from this location:

Enter the specific Search Server address and click Connect. You can view and manipulate the indices that have been defined in Search Server:

One can also use REST Web Services Utility such as Postman to view content from Search Server;

Go into Postman and enter the following URL: http://[NAME OF SEARCH SERVER HOST]:9012/_alias

The following shows a sample result, where Search Server does not have its indexes established yet:

After the indexes are created by running pdm_es_initial_load.bat from the Search Server's default location of C:\Program Files\CA\SC\CASearchServer\search\bin, the same action in Postman will show:

The index names that appear will contain an integer value, which represents the date stamp of the index in UNIX time. The above index circled in red, "sdm_index_cnt_1686248460" has a UNIX time of 1686248460 or Thu Jun 08 2023 18:21:00 GMT+0000.

Additional Information

The following articles discuss how to enable tracing for the related Service Management components as well as re-do indexes in Search Server:

KB Article 12894: Enable CA Service Catalog's TRACE (or DEBUG) level for diagnostic purposes

KB Article 45002: Turning on logging for xFlow

KB Article 189262: How to recreate indexes on CA Search Server?

KB Article 267707: Manual rebuild of Search Server indexes

The following is a selection of KB Articles which discuss various issues that have arisen as a result of Service Point failures due to Search Server functionality needing to be addressed.

KB Article 205022: Unable to locate Service catalog offering under service point.

KB Article 267599: Search Server indexes in a cluster are not being generated

KB Article 220855: Service Point is not updating service offerings from Catalog

KB Article 254742: Service Catalog Offerings Are Not Showing in Service Point

KB Article 215408: Service Point Mobile App and Service Catalog offerings

KB Article 222009: Service Point does not load Service Catalog Offerings after clicking on More button

KB Article 265266: Service Point does not load Sevice Catalog offerings when using SAML Authentication

KB Article 141743: Service Point does not show/find Catalog offerings for Employee role only and Error "Error fetching permissions for" appears in searchMS.log