VSAN Tabs/options not visible post vCenter Upgrade from 7.x to 8.x due to duplicate applmgmt service registrations

Products

• VMware vCenter Server

Issue/Introduction

vSAN menus missing from the vCenter UI

From /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log:

[ERROR] p-nio-127.0.0.1-5090-exec-15 ######## 100005 ###### com.vmware.vise.mvc.controllers.PluginServiceController       A general error occurred while evaluating plugin (com.vmware.vsan.client:8.0.203.10000:-#########) dynamic extensions info from the vSphere Client platform. com.vmware.vapi.std.errors.OperationNotFound: OperationNotFound (com.vmware.vapi.std.errors.operation_not_found) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = apigw.error.serviceIdNotFound,
    defaultMessage = Service not found: com.vmware.vcenter.system_config.feature_state,
    args = [],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = OPERATION_NOT_FOUND

 

Following onto the apigw logs in same location:

 [INFO ] p-nio-127.0.0.1-5090-exec-81 70016595 100273 200023 ApiGatewaySessionManagerImpl [] Pinged session: 200001
 [WARN ] p-nio-127.0.0.1-5090-exec-81 70016595 100273 200023 RoutingTable [mg825bfv-######-auto-####-h5:70016595] Cannot route com.vmware.vcenter.system_config.feature_state.get because service com.vmware.vcenter.system_config.feature_state is not found

 

Duplicate applmgmt service registrations present on vCenter 

To check:

Export service registration details to a text file:

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert > serviceregs.txt

grep "Service Type: applmgmt" serviceregs.txt

Example issue output from a single VC:

# grep "Service Type: applmgmt" serviceregs.txt
        Service Type: applmgmt
        Service Type: applmgmt
        Service Type: applmgmt
        Service Type: applmgmt

 

Note: If there are multiple vCenters in ELM, there will be an applmgmt service registration for each vCenter. If there are more applmgmt service registrations than the number of vCenters in ELM paired with the above symptoms, then this KB may be a match

Environment

vCenter 8.x

Cause

Duplicate applmgmt service registrations causing apigw service to fail to get it's required methods to display the vSAN menus

Resolution

Use lsdoctor tool to remove the extra applmgmt service registrations

1) Download and put the lsdoctor tool onto the vCenter from KB lsdoctor tool

2) As stated in the lsdoctor tool KB - ensure offline snapshots of all vCenters in the same SSO domain are taken as backup

3) Run lsdoctor as described in the KB specifically with the --rebuild option

4) Select option 3 "Replace individual service"

5) A list will appear - select the number that corresponds with applmgmt

6) lsdoctor will remove all instances of applmgmt service registration and then create a new one

7) Restart all services on the VC to apply the change

 

Note: if the lsdoctor doesn't have the current template for service registrations - it'll ask you to select one. Select the one which has the closest build number