lookupsvc service fails to start with error "Problem with directory [/usr/lib/commonlib]"

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

lookupsvc service fails to start with the below log snippets.

/var/log/vmware/vmon/vmon.log

YYYY-MM-DDTHH:MM:SS.189Z Wa(03) host-1918 <lookupsvc> Service exited. Exit code 0
YYYY-MM-DDTHH:MM:SS.189Z Wa(03) host-1918 [ReadSvcSubStartupData] No startup information from lookupsvc.

/var/log/vmware/lookupsvc/tomcat/catalina.log

YYYY-MM-DD HH:MM:SS.148 WARNING [main] org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with directory [/usr/lib/commonlib], exists: [false], isDirectory: [false], canRead: [false]
YYYY-MM-DD HH:MM:SS.161 SEVERE [main] org.apache.catalina.startup.Bootstrap.main Init exception java.lang.ClassNotFoundException: org.apache.catalina.startup.Catalina
                 at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
                 at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
                 at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
                 at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:261)
                 at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:442)

Cause

File corruption in the library files located at /var/opt/apache-tomcat9/lib/ (catalina.jar and ecj-4.20.jar).

Resolution

The below steps are to replace the corrupted jar files located at /var/opt/apache-tomcat9/lib/

1. Take a snapshot of the vCenter appliance.
   
   • If nodes are in Enhanced Linked Mode, power off all vCenters before taking snapshots.
   • Refer ELM snapshots
2. Back up the existing jar files on vCenter
   
   • cp /var/opt/apache-tomcat9/lib/catalina.jar /var/core/catalina.jar.backup
   • cp /var/opt/apache-tomcat9/lib/ecj-4.20.jar /var/core/ecj-4.20.jar.backup
3. Copy the library files from a vCenter with the same build version to the appliance (use WinSCP, scp, or another secure file-transfer tool). Overwrite the originals:
   
   • /var/opt/apache-tomcat9/lib/catalina.jar
   • /var/opt/apache-tomcat9/lib/ecj-4.20.jar
4. Restart all vCenter services:
   
   • service-control --stop --all && service-control --start --all

Note: If necessary, replace all files in the lib folder (/var/opt/apache-tomcat9/lib) with files from the same build version.