vLCM service does not start after patch failure

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

• following an unsuccessful attempt to patch vCenter Server, the vCenter Lifecycle Manager Service fails to start
• /var/log/vmware/vlcm/vlcm.log on the vCenter shows the following errors:

YY-MM-DDTHH:MM:SS  info vlcm [ndu/upgradeorchestrator.go:85] Initializing NDU Orchestrator
YY-MM-DDTHH:MM:SS  info vlcm [ndu/upgradeorchestrator.go:147] Reloading upgrade context from database
YY-MM-DDTHH:MM:SS  error vlcm [dataproviders/upgradecontextpostgre.go:206] Failed to fetch UpgradeContext &database.PostgreReadingError{Text:"failed to perform select", Err:(*pq.Error)(0xc0001bf560)}
YY-MM-DDTHH:MM:SS  error vlcm [ndu/upgradeorchestrator.go:149] Failed to reload from storage previous upgrade.
YY-MM-DDTHH:MM:SS  fatal vlcm [cmd/main.go:225] Unable to start NDU Public Orchestrator Failed to read with error: failed to perform select, underlying error: pq: relation "nduupgrade" does not exist
YY-MM-DDTHH:MM:SS  info vlcm [cmd/main.go:155] Initializing vLCM Service. pid=1042
YY-MM-DDTHH:MM:SS  info vlcm [serviceconfig/config.go:130] Extracting properties from /etc/vmware-vlcm/vlcm_db/vlcm.properties
YY-MM-DDTHH:MM:SS  info vlcm [serviceconfig/config.go:91] Getting rhttpproxy port for ServiceConfig
YY-MM-DDTHH:MM:SS  info vlcm [serviceconfig/config.go:98] Returned rhttpproxy port number is 443
YY-MM-DDTHH:MM:SS  info vlcm [serviceconfig/config.go:239] Extracting vlcm plugin version info from /etc/vmware-vlcm/vlcm_version_info.yaml
YY-MM-DDTHH:MM:SS  info vlcm [logger/teelogger.go:54] [opID=vapi] Message formatter created for en, en, UTC, 2, SHORT_DATE_TIME
YY-MM-DDTHH:MM:SS  info vlcm [logger/teelogger.go:47] [opID=vapi] Supported languages are: [en de es fr ja ko zh-CN zh-TW]
YY-MM-DDTHH:MM:SS  info vlcm [logger/teelogger.go:54] [opID=vapi] Message formatter created for en, en, UTC, 2, SHORT_DATE_TIME
YY-MM-DDTHH:MM:SS  info vlcm [database/connection.go:50] Connected to VCDB!
YY-MM-DDTHH:MM:SS  info vlcm [cmd/main.go:206] Starting NDU Public Orchestrator.
YY-MM-DDTHH:MM:SS  info vlcm [ndu/upgradeorchestrator.go:85] Initializing NDU Orchestrator
YY-MM-DDTHH:MM:SS  info vlcm [ndu/upgradeorchestrator.go:147] Reloading upgrade context from database
YY-MM-DDTHH:MM:SS  error vlcm [dataproviders/upgradecontextpostgre.go:206] Failed to fetch UpgradeContext &database.PostgreReadingError{Text:"failed to perform select", Err:(*pq.Error)(0xc000206000)}
YY-MM-DDTHH:MM:SS  error vlcm [ndu/upgradeorchestrator.go:149] Failed to reload from storage previous upgrade.
YY-MM-DDTHH:MM:SS  fatal vlcm [cmd/main.go:225] Unable to start NDU Public Orchestrator Failed to read with error: failed to perform select, underlying error: pq: relation "nduupgrade" does not exist

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

Previous entries in the lookupsvc from old VCSA/PSC.
WCP service is running as user wcp instead of root. 
WCP service not running

Resolution

Before attempting the steps outlined below, please make sure to create an offline snapshot of the vCenter Server Appliance (in powered off state). If the affected VCSA is a member of an Enhanced Linked Mode (ELM) replication setup, please ensure to have offline snapshots of all other ELM nodes as well.

1. Download and copy the attached script vlcm_db_upgrade.sql onto the VC.
2. SSH into the VC.
3. Make sure vlcm is stopped:

# service-control --stop vlcm

1. Grab the db password for vlcmuser from `/etc/vmware-vlcm/vlcm_db/vlcm.properties:

# cat /etc/vmware-vlcm/vlcm_db/vlcm.properties | grep "password ="

password = XXXXXXXXXXX

5. Execute this script by passing the password in the `PGPASSWORD` env var (enclosed in ''):

$ PGPASSWORD='Password_here' psql -p 5432 -U vlcmuser -d VCDB -f path/to/the/script.sql

6. Start vlcm service:

$ service-control --start vlcm

7. Check VMdird status with vdcadmintool:

echo 6| /usr/lib/vmware-vmdir/bin/vdcadmintool

8. If the status of VMdird is standalone, then run the following command:

For any version below 8.0 U3:
/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif"

For 8.0 U3 and above versions:
/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif -L /var/log/vmware/vmdird/vmdird.log"

9. Restart the VCSA only if VMdird state was standalone and you applied step 8.

Additional Information