After upgrade from version 7.x to 8.x, vCenter version for endpoint shows as 7.0 instead of 8.0 in lstool output

Products

• VMware vCenter Server

Issue/Introduction

• At one time during upgrade, replication was not functioning.
• Version reported for endpoints shows as "Version: 7.0" instead of "Version: 8.0"
• vCenters are in Enhanced Link Mode (ELM)
• Output of :/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --type vcenterserver

Shows as "Version: 7.0" for some or all vCenter endpoint nodes, example:

        Name: AboutInfo.vpx.name
        Description: AboutInfo.vpx.name
        Service Product: com.vmware.cis
        Service Type: vcenterserver
        Service ID: <vcenter UUID>
        Node ID: <vCenter Node ID>
        Owner ID: vpxd-<endpointID>@vsphere.local
        Version: 7.0
        Endpoints:
                Type: com.vmware.cis.data.provider
                Protocol: vmomi
                URL: https://<vcenterFQDN>:443/sdk

Environment

vCenter Server 8.x

Cause

This is a result of replication failure prior to upgrade.  The vCenter nodes were not updating the version values on all the remaining nodes during an upgrade and caused a mismatch of versions within the endpoints.

Once replication is fixed, the nodes are in sync, but some of the version numbers may be incorrect.

Resolution

A rebuild of vCenter endpoints is needed to fix the mismatch version of the endpoint.  This can be accomplished by using the lsdoctor tool.

1. Power down all vCenter servers in the ELM configuration.
   
   
2. Create a snapshot of each vCenter server.
   
   
3. Power up all vCenter servers and wait for services to all start.
   
   
4. On vCenter that is showing incorrect version number, login via ssh and run lsdoctor - See KB: https://knowledge.broadcom.com/external/article?articleId=320837
   • Follow lsdoctor installation instructions to upload and unzip lsdoctor on the vCenter appliance
     
     
5. Within directory of extracted lsdoctor files:
   • Run the command:  python lsdoctor -r
   • Use Option 2 to rebuild all services.
   • Follow the prompts to enter password and rebuild services for that node.
     
     
6. Once rebuild is complete, restart all services using
   • service-control --stop --all
   • service-control --start --all
     
     
7. Once services have restarted, continue onto any other vCenters that are showing incorrect version numbers and perform steps 4-6.

Additional Information

This usually doesn't cause any actual issue, but can indicate a replication issue was present at one point and the resolution of this article should fix the version problem as it will regenerate all endpoints for the given vCenter.