vCenter upgrade fails at Stage 2 step 2 with error "unable to find the sso endpoint for reregistering"

Products

• VMware vCenter Server

Issue/Introduction

While upgrading vCenter server, a UI error is shown in Stage 2, Step 2  similar to:

"/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 281, in boot self.reregisterSTSUsing LookupService() File "/usr/lib/vmidentity/firstboot/vmidentity firstboot.py", line 435, in reregisterSTSUsingLookupService raise e File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 420, in reregister STSUsing LookupService raise Exception 'Unable to find the sso endpoint for reregistering) Exception: Unable to find the sso endpoint for reregistering"

You may notice that attempting to login to this vCenter redirects the user to another vCenter in linked mode for login.

Environment

VMware vCenter Server 7+

Enhanced Linked Mode

Cause

This vCenter is missing between 1 and 4 service registrations all responsible for the SSO service:  cs.identity, sso:admin, sso:sts, and sso:groupcheck.  The reason for these services having gone missing is unknown.

To identify this issue, run the following commands.  You should see a count equal to the number of vCenter servers in linked mode (2 vCenter servers in the below example):

Expected:

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://$HOSTNAME/lookupservice/sdk --type cs.identity | grep "Service Type" | sort | uniq -c
      2         Service Type: cs.identity

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://$HOSTNAME/lookupservice/sdk --type sso:admin | grep "Service Type" | sort | uniq -c
      2         Service Type: sso:admin

 

Problematic:

 

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://$HOSTNAME/lookupservice/sdk --type cs.identity | grep "Service Type" | sort | uniq -c
      1         Service Type: cs.identity

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://$HOSTNAME/lookupservice/sdk --type sso:admin | grep "Service Type" | sort | uniq -c
      1         Service Type: sso:admin

 

These service registrations are necessary for normal functionality in enhanced linked mode, especially during upgrade.

Resolution

NOTE:  Please ensure you have proper snapshots and/or backups before performing these steps!  For more details see VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

 

1. Download and unzip the lsdoctor tool as outlined in Using the 'lsdoctor' Tool
2. From the lsdoctor tool directory, run the following commands:
   1. Set the classpath:

      1. CLASSPATH=$(python lib/lsdoctor_defaults.py)

   2. Export the machine certificate:

      1. /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT > machine.crt

   3. Recreate the cs.identity endpoint:

      1. /usr/java/jre-vmware/bin/java -cp $CLASSPATH -Dlog4j.configurationFile=/usr/lib/vmware-lookupsvc/conf/initls-log4j2.xml -Dvmware.log.dir=/var/log/vmware/lsdoctor com.vmware.vim.lookup.tools.InitializeLookupService --cert-path "machine.crt" --host-name "$HOSTNAME" --http-port 443

   4. Recreate the legacy endpoints:

      1. /usr/java/jre-vmware/bin/java -cp $CLASSPATH -Dlog4j.configurationFile=/usr/lib/vmware-lookupsvc/conf/initls-log4j2.xml -Dvmware.log.dir=/var/log/vmware/lsdoctor com.vmware.vim.lookup.tools.InitializeLookupService --cert-path "machine.crt" --host-name "$HOSTNAME" --http-port 443 --legacy

3. Restart all services:

   1. service-control --stop --all && service-control --start --all

4. Retry the upgrade.