Error "vCenter upgrade failed at the VCENTER_UPGRADE_SWITCHOVER stage." during vCenter upgrade from 8.0.3.1000 to 9.1.1.0

Products

• VMware SDDC Manager / VCF Installer
• VMware vCenter Server

Issue/Introduction

• vCenter upgrade from 8.0.3.1000 to 9.1.1 fails during the "VCENTER_UPGRADE_SWITCHOVER stage" when performing the upgrade through SDDC manager/VCF Operations.
• While checking /storage/log/vmware/vlcm/vlcm.log on the target vCenter appliance:

      ERROR vlcm 26849 [vc@4413] [exec/exec.go:126] [ObjectID=118] Run command [/usr/bin/service-control --start --all] failed (duration: 3m16.684s)
      ERROR vlcm 26849 [vc@4413] [logger/multiline.go:19] [ObjectID=118] service-control failed
      ERROR vlcm 26849 [vc@4413] [logger/multiline.go:45] [ObjectID=118] ERR:001 exit status 1
      ERROR vlcm 26849 [vc@4413] [logger/multiline.go:53] [ObjectID=118] ERR:002-
      ERROR vlcm 26849 [vc@4413] [logger/multiline.go:45] [ObjectID=118] OUT:001 Service-control failed. Error: Failed to start services in profile ALL. RC=4, stderr=Failed to start content-library services. Error: A system error occurred. Check logs for details

• Content Library service firstboot fails and the target appliance rolls back and gets deleted as part of the RDU method of upgrade.
• /storage/log/vmware/content-librarycls-pre-start.log shows as below:

      cls-pre-start.log:[INFO] Re-registering content library service
      cls-pre-start.log:[INFO] Reregistering CL service (ID: 0609f2f2-####-###-###-###) with lookup service
      cls-pre-start.log:[ERROR]Failed to re-register content library. (lookup.fault.EntryNotFoundFault) {
      cls-pre-start.log:[INFO] Creating content-library service account creds.
      cls-pre-start.log:[INFO] Found service account VSR file for content-library-user. Fetching service account values from the file.

 

Environment

VMware vCenter Server 8.x

VMware vCenter Server 9.x

VCF 9.x

Cause

The content-library service registration (cis.cls) is missing on the source vCenter Server.

Resolution

To resolve the issue:

             1. Take offline snapshot of both the vCenters , as they are in linked mode.

             2. Download LSDOCTOR TOOL

             3. Unzip command: python -m zipfile -e lsdoctor-260716.zip .

             4. Using -r -> Option 3 to register the individual service (cis.cls)  and use json template version listed which is more closer to the target vCenter version.( eg 25417926_9.1.0.0100_vcsa.json)

             5. Restart the content library service using the command "service-control --stop content-library && service-control --start content-library"