VSAN Tabs/options not visible post vCenter Upgrade from 7.x to 8.x

Products

• VMware vSAN

Issue/Introduction

• Post vCenter Server Upgrade version from 7.x to 8.x vSAN tabs and options are missing from UI.
• vCenter Server rebooted, along with services restarted, but the vSAN option under Cluster->Monitor and Configurations are still missing from vSphere UI.

Symptoms :

The below logs shows authentication issues,

vSphere vCenter /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log:

[2024-10-24T05:47:06.779Z] [ERROR] tp-nio-127.0.0.1-5090-exec-5 70000225 100007 200006 com.vmware.vise.mvc.controllers.PluginServiceController A general error occurred while evaluating plugin (com.vmware.vsan.client:8.0.203.10000:1168321836) dynamic extensions info from the vSphere Client platform. com.vmware.vcenter.apigw.security.AuthenticationException: Failed to log into [uri=http://localhost:8201/api, sessionMgr=SessionManagerInfo [_sessionMgrSvcId=com.vmware.cis.session, _loginOpId=create, _logoutOpId=delete], ssoDomain=vsphere.local(e7659782-57be-4bd6-82d6-a2e37bc86275)]: com.vmware.vapi.std.errors.unauthenticated => {data=<unset>, error_type=UNAUTHENTICATED,messages=[com.vmware.vapi.std.localizable_message => {args=[], default_message=Unable to authenticate user, localized=<unset>, id=vapi.security.authentication.invalid, params=<unset>}]}
        at java.lang.Thread.getStackTrace(Thread.java:1564)
        at com.vmware.vapi.bindings.client.AsyncCallbackSyncAdapter.get_aroundBody1$advice(AsyncCallbackSyncAdapter.java:49)
        at com.vmware.vapi.bindings.client.AsyncCallbackSyncAdapter.get(AsyncCallbackSyncAdapter.java:1)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:145)
        at com.vmware.vcenter.system_config.FeatureStateStub.get(FeatureStateStub.java:48)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
        at com.sun.proxy.$Proxy408.get(Unknown Source)
        at com.vmware.vise.plugin.filter.feature.ConditionalExtensionFeatureStatesFilter.retrieveVcFeatureStateSwitches(ConditionalExtensionFeatureStatesFilter.java:100)

From Appliance management: /var/log/vmware/service log

Traceback (most recent call last):
  File "/usr/lib/applmgmt/vapi/py/vmware/appliance/vapi/auth.py", line 252, in authenticate
    username = token.username
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 487, in username return self.get_name_id().value
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 983, in get_name_id '//saml2:Subject/saml2:NameID', self.reference)
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 478, in reference self.validate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 1213, in validate reference = super(HolderOfKeyToken, self).validate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 509, in validate self.validate_certificate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 709, in validate_certificate raise AuthenticationError(vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.
2024-10-20T19:00:06 PM UTC [10093]DEBUG:vmware.appliance.extensions.authentication.authentication_sso:Downloading trusted certs from url : http://localhost:7080/idm/tenant/vsphere.local/certificates?scope=TENANT
2024-10-20T19:00:06 PM UTC [10093]DEBUG:vmware.appliance.extensions.authentication.authentication_sso:Downloading trusted certs from url : http://localhost:7080/idm/tenant/vsphere.local/certificates?scope=TENANT
2024-10-20T19:00:06 PM UTC [10093]ERROR:vmware.appliance.vapi.auth:Could not parse HOK Token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 507, in validate self.validate_certificate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 709, in validate_certificate raise AuthenticationError(vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.

 

Environment

VMware VSAN 8.x

Cause

• The above functions loads certificates for x509 and trust certificates, this is from VC. Using this it performed verification. This Exception "One or more certificates cannot be verified.". is raised if one or more certs from the certificate Chain cannot be verified.

• If there are multiple STS certificates in VC with expired, that also causes same issue.

Resolution

Below Steps to be followed to fix issue:

• First, Run checksts.py to check how many certificates available, If the STS certificate is about to expire or if it is already expired                                                                                                                                                                                                                                                                                                                                                                   Refer article for sts file - Checking Expiration of STS Certificate on vCenter Servers
• If there are multiple certificates please run the script fixsts.py                                                                                                                                                                                                                                                                                                                                                                                                                                                                          To resolve the Signing certificate is not valid error:

1. Download the attached fixsts.sh script from this article and upload to the impacted PSC or vCenter Server with Embedded PSC to the /tmp folder.
2. If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter:

   # chsh -s /bin/bash

3. Connect to the PSC or vCenter Server with an SSH session if you have not already per Step 2.
4. Navigate to the /tmp directory:

   # cd /tmp

5. make the file executable:

   # chmod +x fixsts.sh

6. Run the script:

   # ./fixsts.sh

7. Restart services on all vCenters and/or PSCs in your SSO domain by using below commands:

   # service-control --stop --all && service-control --start --all

Refer the article for sts file - "Signing certificate is not valid" error in vCenter Server Appliance

Note: Please Capture the VC snapshots before performing any change activity. If the VC in linked mode capture offline Snapshots.