"Failure to register SyncaaS tables in postgressql", vCenter upgrade/patching fails while starting 'lookupsvc' service

Products

• VMware vCenter Server

Issue/Introduction

• Attempting to upgrade vCenter Server from version 7 to to 8.0 fails while starting lookupsvc.
  
  
  
  
• Same failure is observed while patching 8.0 to higher builds, Patchrunner log file will show below errors:
  
  /var/log/vmware/applmgmt/Patchrunner.log

YYYY-MM-DDTHH:MM:SSZ lookupsvc:Patch INFO patch_01 YYYY-MM-DDTHH:MM:SSZ  ERROR starting lookupsvc rc: 4, stdout: , stderr: Start service request failed. Error: A system error occurred. Check logs for details
YYYY-MM-DDTHH:MM:SSZ  Lookup Service firstboot failed
YYYY-MM-DDTHH:MM:SSZ lookupsvc:Patch INFO patch_01
  File "/storage/seat/software-update14jkqkjp/stage/scripts/patches/payload/components-script/lookupsvc/patches/patch_01.py", line 40, in doPatching
    raise Exception("Lookupsvc Patching failed to execute firstboot operation")
Exception: Lookupsvc Patching failed to execute firstboot operation
YYYY-MM-DDTHH:MM:SSZ ERROR __main__ Patch vCSA failed
  
  
  
• vMon logs will show failure while modifying DB tables
  
  /var/log/vmware/vmon/vmon.log

[YYYY-MM-DDTHH:MM:SS] In(05) host-1234 Received start request for lookupsvc
[YYYY-MM-DDTHH:MM:SS] In(05) host-1234 <lookupsvc-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-lookupsvc/prestart.py /var/log/vmware/lookupsvc/lookupsvc-prestart.log
[YYYY-MM-DDTHH:MM:SS] Wa(03) host-1234 <lookupsvc> Service pre-start command's stderr: Traceback (most recent call last):
[YYYY-MM-DDTHH:MM:SS] Wa(03)+ host-1234   File "/usr/lib/vmware-lookupsvc/prestart.py", line 276, in <module>
[YYYY-MM-DDTHH:MM:SS] Wa(03)+ host-1234
[YYYY-MM-DDTHH:MM:SS] Wa(03) host-1234 <lookupsvc> Service pre-start command's stderr:     raise e
[YYYY-MM-DDTHH:MM:SS] Wa(03)+ host-1234   File "/usr/lib/vmware-lookupsvc/prestart.py", line 272, in <module>
[YYYY-MM-DDTHH:MM:SS] Wa(03)+ host-1234
[YYYY-MM-DDTHH:MM:SS] Wa(03) host-1234 <lookupsvc> Service pre-start command's stderr:     grant_permission_to_VCDB_role_for_lookupsvc()
[YYYY-MM-DDTHH:MM:SS] Wa(03)+ host-1234   File "/usr/lib/vmware-lookupsvc/prestart.py", line 55, in grant_permission_to_VCDB_role_for_lookupsvc
[YYYY-MM-DDTHH:MM:SS] Wa(03)+ host-1234
[YYYY-MM-DDTHH:MM:SS] Wa(03) host-1234 <lookupsvc> Service pre-start command's stderr:     _raise_error("Failure granting permissions to lookupsvc VCDB role.", command, rc, stdout, stderr)
[YYYY-MM-DDTHH:MM:SS] Wa(03)+ host-1234
[YYYY-MM-DDTHH:MM:SS] Wa(03) host-1234 <lookupsvc> Service pre-start command's stderr: NameError: name 'command' is not defined

• Lookupservice pre-start logs will show below errors :
  
  /var/log/vmware/lookupsvc/lookupsvc-prestart.log

YYYY-MM-DDTHH:MM:SSZ ERROR vpg_sync_regiatry.Py --operation install --database VCDB exited with return code [1]
YYYY-MM-DDTHH:MM:SSZ ERROR : during Generation of the diff: Invalid identifier: "vcReadonly"
YYYY-MM-DDTHH:MM:SSZ  ERROR : Failure to register Syncaas tables in postgressql

Environment

VMware vCenter 7.x

VMware vCenter 8.x

Cause

This issue is caused when Capital letters are used to name the Custom roles created in vCenter Database (VCDB)

Resolution

1. Revert back to original vCenter Server if the vCenter Server is already in failed state.
2. Login to the VCSA using SSH
3. Connect to the VCDB using 'psql' command:
   
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
   
   
4. Rename the role name shown in the error message to all lowercase
   
   ALTER ROLE "RoleName_with_CapitalLetters" RENAME TO "rolename_in_lowercase";
   
   Example - ALTER ROLE "vcReadonly" RENAME TO "vcreadonly";
   
   
5. Exit from VCDB using below command:
   
   \q
   
   
6. Re-run the upgrade or patching