Cluster Hardware Compatibility check or Update Image Compliance checks fail with error "An internal error occurred during execution"

Products

• VMware vSphere ESXi

Issue/Introduction

SYMPTOM:  

• When a vCenter Cluster's hardware compatibility 'run checks' or cluster Update Image compliance check/pre-checks tasks are run, a failure message "An internal error occurred during execution" is returned.
  
        Web UI Error Message:
       

Environment

vCenter 7.0.x

Cause

The caching database file used for the hcl task has become corrupted.

• Investigation Details:
  • Path+File with SQLite error corruption message:
    /var/log/vmware/vmware-updatemgr/vum-server/hcl_python_lib.log
  • The hcl_python_lib.log file contain traceback message(s) reporting "database disk image is malformed" similar to the following message: 

Traceback (most recent call last):
  File "/usr/lib/vmware-updatemgr/python/hcl/hcl.py", line 154, in main
    result = callback(**callbackArgs)
  File "/usr/lib/vmware-updatemgr/python/hcl/report/vsan.py", line 173, in generateVsanHclConstraints
    result = generateHclConstraints(hostId, targetVersion, vSanHclConstraints,
  File "/usr/lib/vmware-updatemgr/python/hcl/report/hcl.py", line 182, in generateHclConstraints
    hardware = hostHardwareDiscovery.getHardware(vSanHclConstraints)
  File "/usr/lib/vmware-updatemgr/python/hcl/hardware_discovery/discoverer.py", line 78, in inner
    return func(*args, **kwargs)  # pylint: disable=E1102
  File "/usr/lib/vmware-updatemgr/python/hcl/hardware_discovery/discoverer.py", line 830, in getHardware
    vsanControllers = self._collectVsanControllers()
  File "/usr/lib/vmware-updatemgr/python/hcl/hardware_discovery/discoverer.py", line 509, in _collectVsanControllers
    cacheResult = getCacheFactory().getVsanControllersCache() \
  File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/cache/vsan_controllers_cache.py", line 50, in get
    self.executor.execute("DELETE FROM VsanControllersCache WHERE (? - created_at) > ?",
  File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/cache/executor.py", line 86, in execute
    c.execute(statement, args)
sqlite3.DatabaseError: database disk image is malformed

Resolution

Fix:

Remove the corrupted hcl cache database file from the vCenter Server Appliance (VCSA).  The file will be recreated.

• NOTE:  There is no critical vCenter data contained in the vsan_hcl_cache.db file. 
  The DB files is used only to cache the results of the HCL system hardware check.

Process to remove the hcl cache database file:

• Optional:  Before making the change below, take a VCSA VM snapshot and VAMI backups of the VCSA.
  While this is a 'low' risk change to the VCSA, the VCSA is often rated as a 'high' impact for changes.
  Take a VCSA VM snapshot/ VAMI backup before this change as warranted for the impacted users and related systems.
  
  
• Open a console/ssh session to the impacted VCSA. Logon with the 'root' user ID and password.
  
  
• Delete the hcl SQLight cache database file by running the command below:
       rm /storage/updatemgr/hcl/vsan_hcl_cache.db
• Restart the VC Services from the command line or reboot the VCSA appliance to restart the vCenter services after the file has been removed.
  
  • Command line syntax to restart vCenter services:
    • Stop All VCSA Services:
        service-control --stop --all 
    • Start All VCSA Services
          service-control--start --all 

• Re-run the update task(s) to check Hardware Compliance or Update Image compliance and verify the issue is resolved. The HCL cache database file will be recreated when the tasks are run again.
  
  
• Cleanup Item: Remove the VCSA snapshot after checkout has confirmed the issue is resolved.