0015816: search index cannot be regenerated

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

Following the steps in this article, the search index rebuild fails for a customer:
https://knowledge.broadcom.com/external/article/145760/awa-122x-123x-how-to-rebuild-search-ind.html

JCP log:

The indexbuilder fails upon updating the search index, and running in loops.
Also the following error occurs in the logs:

U00045014 Exception 'java.lang.NullPointerException: "null"' at 'com.automic.persistence.api.FullTextSearch$ObjectHead.<init>():112'.

U00003620 Routine 'IndexBuilder doing a rollback on last lucene commits..' forces trace because of error.
U00003450 The TRACE file was opened with the switches '0000000000000000'.
U00003449 Output to the TRACE file is finished.
Last lucene commits were rolled back!
U00003434 Server routine 'IndexBuilder/TIMER' required '1' minutes and '2' seconds for processing.
U00045135 Search index update: failed to determine time-stamp of last search index update. Rebuilding entire index.
U00045136 Search index update: Detected 77824 objects which need to be re-indexed. 1057 objects are being removed. (3,453s)
U00045133 Search index update: executing 2 partition(s) for 1057 deleted object(s).
U00045134 Search index update: partition #0 finished (0,000s).
U00045134 Search index update: partition #1 finished (0,000s).
U00045122 Search index update: updated 1057 objects (3,453s)
U00045133 Search index update: executing 1 partition(s) for 136 modified calendar event object(s).
U00045134 Search index update: partition #0 finished (0,547s).
U00045122 Search index update: updated 136 objects (4,000s)
U00045133 Search index update: executing 78 partition(s) for 77824 modified object(s).
U00045134 Search index update: partition #0 finished (0,891s).
...
U00045134 Search index update: partition #53 finished (0,875s).
U00003620 Routine 'IndexBuilder doing a rollback on last lucene commits..' forces trace because of error.
U00003450 The TRACE file was opened with the switches '0000000000000000'.
U00003449 Output to the TRACE file is finished.
Last lucene commits were rolled back!

We have checked the JCP trace files, and the following entries were generated along with the Java exception:

This is related to a custom attribute within the system, which was deleted since then:

SELECT OCA_OH_Idnr, OCA_NAME, OCA_VALUE FROM OCA WHERE OCA_OH_Idnr in (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
UCUDB32 SLCT RET 0000 HSTMT: 0000000000000000 VALUE: 0000000000000000 ALL: 0,00000 DB: 0.00000 ODBC: 0.00000 UDB: 0.00000
UCUDB32 READ RET 0000 HSTMT: 0000000000000000 VALUE: 0000000000000001 ALL: 0,00000 DB: 0.00000 ODBC: 0.00000 UDB: 0.00000
U00009909 TRACE:(DB-DATEN: OCA_OH_IDNR) size 000007
00000000 31303031 343339 >1001439<
U00009909 TRACE:(DB-DATEN: OCA_NAME) size 000012
00000000 26646570 6172746D 656E7423 >&department#<
U00009909 TRACE:(DB-DATEN: OCA_VALUE) size 000010
00000000 4163636F 756E7469 6E67 >Accounting<
UCUDB32 READ RET 0000 HSTMT: 0000000000000000 VALUE: 0000000000000001 ALL: 0,00000 DB: 0.00000 ODBC: 0.00000 UDB: 0.00000
U00009909 TRACE:(DB-DATEN: OCA_OH_IDNR) size 000007
00000000 31303031 343339 >1001439<
U00009909 TRACE:(DB-DATEN: OCA_NAME) size 000012
00000000 26646570 6172746D 656E7423 >&department#<
U00009909 TRACE:(DB-DATEN: OCA_VALUE) size 000011
U00000000 44657665 6C6F706D 656E74 >Development<
UCUDB32 READ RET 0001 HSTMT: 0000000000000000 VALUE: 0000000000000000
UCUDB32 CLST RET 0000 HSTMT: 0000000000000000 VALUE: 0000000000000000
U00045014 Exception 'java.lang.NullPointerException: "null"' at 'com.automic.persistence.api.FullTextSearch$ObjectHead.<init>():112'.
U00003620 Routine 'IndexBuilder doing a rollback on last lucene commits..' forces trace because of error.

Environment

Release : 12.3.x, 21.x

Component: Automic Workload Automation

Cause

Corrupted objects in the database.

The value for OH_OTYPE is ' ' - sign for a blank space, but this should not exist normally.

 

Resolution

Turn on DB=4 and TCP/IP=2 traces and provide the logs and traces to the Support Team.