Index Contents and Documents for Searches Job fails with SRCH-01003 Error

Products

• Clarity PPM On Premise
• Clarity PPM SaaS

Issue/Introduction

Index content for document and searches job fails with 'SRCH-01003: Error occurred while indexing search contents' and 'Lock obtain timed out: [email protected]:\searchindex\clarity\nspider\write.lock' errors in the bg-ca logs.  

Sample Stack trace 

"ERROR 2021-12-06 23:38:16,136 [http-nio-8082-exec-30] xbl.XOGCustomObjectInstanceHandler (clarity:xogadmin:75288047__298B3211-02DF-4F38-8D5B-F8C4FE867A5E:XOG::customObjectInstance::write) exception XOGging object
com.niku.union.odf.exception.ODFException: Error writing file searchindex\clarity\Test.csv to file system"
"Caused by: com.ca.clarity.union.search.engine.SearchException: SRCH-02001: Error while retrieving search contents. Contact your system administrator.
at com.niku.search.engine.lucene.LuceneSearchEngine.addFileToIndex(LuceneSearchEngine.java:123)
at com.niku.odf.object.ODFAbstractObjectImpl.processNewAttachment(ODFAbstractObjectImpl.java:5092)
... 74 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: [email protected]:\searchindex\clarity\nspider\write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:85)"

Cause

The cause can be the lucene engine used for searching is corrupted and you may need to perform additional step to fix it 

Environment

Release: All Supported Clarity Release 
Component: PPMSTU

Resolution

Steps to resolve: 

1.Login to NSA 
2.Under properties > Documents and Search, Disable Use Synchronous Indexing 
3.Restart services 
4.In Clarity > Home > Reports and Jobs 
5.Run the Index contents and documents for searches job. 
6.See if the job completes successfully. 

Note: The Index content for document and searches job cannot be run while the the 'Disable Use Synchronous' option is checked. Either the box should be checked OR the index Content for document and searches job run. 

SaaS customers, please contact Broadcom Support for assistance.

Additional Information

If the above steps dont help then you need to perform below 

***IMPORTANT: Please perform the activity after-hours, as it is very resource-intensive and could impact performance and ensure to stop all the clarity services before performing the steps. For very large filestores, please don't run the command's without contacting Broadcom Support***

1. Pause the "Index contents and documents for search" scheduled job
2. Turn off synchronous indexing in the CSA (CSA  Documents & Search and uncheck Use Synchronous Indexing)
3. Restart the services after doing this.
4. In CSA add "-Djava.io.tmpdir=c:\javatmp" to app service jvm  parameters (unless this parameter is already present and you can set the path based on your operating system)
5. In CSA add "-Djava.io.tmpdir=c:\javatmp" to bg service jvm parameters (unless this parameter is already present and you can set the path based on your operating system)
6. Search for and delete any files that end in write.lock on your app server(s).
   They will look something like this:
   lucene-62db5567fbeaae16396e79fb8fe42895-write.lock
7. Restart all the services.
8. Delete the nspider and nspiderfile folders from the file server  
9. Log into the app server as someone who has full control of the filestore folders including the ability to create new folders/files like the nspiderfiles folders 
10. Run the "admin search recreate-index-data" command 
11. Run the "admin search recreate-index-files" command