Image and Hardware Compatibility tabs missing from the Update tab in an image based cluster, also possible after upgrade to vCenter 8.0.
search cancel

Image and Hardware Compatibility tabs missing from the Update tab in an image based cluster, also possible after upgrade to vCenter 8.0.

book

Article ID: 345083

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  1. After upgrading vCenter server from a 7.x version to a 8.x version the Update tab in an image based cluster is missing 2 objects. (Image and Hardware Compatibility tabs)
  2. You see log entries below: 

var/log/vmware/applmgmt/applmgmt.log:

yyyy-mm-dd [2992]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.vcenter.system_config.feature_state, operation_id: get
yyyy-mm-dd [2992]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:Required privileges = ['ModifyConfiguration']
yyyy-mm-dd [2992]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:User=vsphere.local\vsphere-webclient-####-###-###-####-#####8, groups={'vsphere.local\\Everyone', 'vsphere.local\\SolutionUsers', 'vsphere.local\\ActAsUsers', 'vsphere.local\\Administrators', 'vsphere.local\\LicenseService.Administrators', 'vsphere.local\\vSphereClientSolutionUsers'}
yyyy-mm-dd [2992]DEBUG:root:Validated user privileges in localstore or SSO
yyyy-mm-dd [2992]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:Required privileges = ['ModifyLocalConf']
yyyy-mm-dd [2992]DEBUG:vmware.appliance.extensions.authorization.authorization_sso:User=vsphere.local\vsphere-webclient-####-####-####-###-######8, groups={'vsphere.local\\Everyone', 'vsphere.local\\SolutionUsers', 'vsphere.local\\ActAsUsers', 'vsphere.local\\Administrators', 'vsphere.local\\LicenseService.Administrators', 'vsphere.local\\vSphereClientSolutionUsers'}
yyyy-mm-dd [2992]DEBUG:root:Validated user privileges in localstore or SSO



var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log:

yyyy-mm-dd [ERROR] http-nio-3###-exec-## ####### ##### ##### com.vmware.vise.mvc.controllers.PluginServiceController A general error occurred while evaluating plugin (com.vmware.vlcm.client:8.0.0.21216066:-#####) dynamic extensions info from the vSphere Client platform. com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized) => {
 messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
 id = vapi.security.authorization.invalid,
 defaultMessage = Unable to authorize user,
 args = [],
 params = <null>,
 localized = <null>
 }],
 data = <null>,
 errorType = UNAUTHORIZED
 }
 at java.lang.Thread.getStackTrace(Thread.java:1564)
 at com.vmware.vapi.bindings.client.AsyncCallbackSyncAdapter.get_aroundBody1$advice(AsyncCallbackSyncAdapter.java:49)
 at com.vmware.vapi.bindings.client.AsyncCallbackSyncAdapter.get(AsyncCallbackSyncAdapter.java:1)
 at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:145)
 at com.vmware.vcenter.system_config.FeatureStateStub.get(FeatureStateStub.java:47)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
 at com.sun.proxy.$Proxy387.get(Unknown Source)
 at com.vmware.vise.plugin.filter.feature.ConditionalExtensionFeatureStatesFilter.retrieveVcFeatureStateSwitches(ConditionalExtensionFeatureStatesFilter.java:101)
 at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
 at com.vmware.vise.plugin.filter.feature.ConditionalExtensionFeatureStatesFilter.filter(ConditionalExtensionFeatureStatesFilter.java:68)
 at com.vmware.vise.plugin.filter.impl.PluginDynamicExtensionFilteringServiceImpl.conditionalFiltering(PluginDynamicExtensionFilteringServiceImpl.java:168)







    3.This issue can also be observed if there is an expired STS Tenant 

/var/log/vmware/vmware-updatemgr/vum-server:

yyyy-mm-dd error vmware-vum-server[13336] [Originator@6876 sub=CertManager] [CertManager 230] Retrieved invalid certificate

 

var/log/vmware/applmgmt/applmgmt.log:

yyyy-mm-dd [8618]DEBUG:vmware.appliance.extensions.authentication.authentication_sso:Downloading trusted certs from url : http://localhost:7080/idm/tenant/vsphere.local/certificates?scope=TE NANT 
yyyy-mm-dd [8618]DEBUG:vmware.appliance.extensions.authentication.authentication_sso:Downloading trusted certs from url : http://localhost:7080/idm/tenant/vsphere.local/certificates?scope=TE NANT 
yyyy-mm-dd [8618]ERROR:vmware.appliance.vapi.auth:Could not parse HOK Token Traceback (most recent call last):   File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 507, in validate     self.validate_certificate()   File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 709, in validate_certificate     raise AuthenticationError( vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.

Environment

VMware vCenter Server 8.0

Cause

  • This behavior is caused by the SystemConfiguration.Administrators group missing for the token on the environment Or if the SystemConfiguration.Administrators group is present an expired STS Tenant can also cause this behaviour.



Resolution

  1. The token can be added back by performing the steps below:
    1. Take proper snapshots of the environment. (if the VC is member of Enhanced linked mode, please take powered off snapshots from all nodes)
    2. Navigate to Administration -> Single Sign On -> Users and Groups
    3. Select Groups and search for SystemConfiguration.Administrators
    4. Select Add Members
    5. Search and select Administrators
    6. Then click save.
    7. Log out and log back in.
  2. To resolve the Expired STS Tenant
    1. Download the attached fixsts.sh script from article FIXSTS and upload to the impacted vCenter Server with Embedded PSC to the /tmp folder.
    2. If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter:
      #chsh -s /bin/bash
  3. Connect to the PSC or vCenter Server with an SSH session if you have not already
  4. Navigate to the /tmp directory:
     
    # cd /tmp

     

  5. Make the file executable:
     
     #chmod +x fixsts.sh
  6. Run the script:
     
    # ./fixsts.sh
  7. Restart services on all vCenter's in your SSO domain by using below commands:
     
    # service-control --stop --all && service-control --start --all

    Note: Restart of services will fail if there are other expired certificates like Machine SSL or Solution User Using vSphere Certificate Manager to Replace SSL Certificates

  8. The script will ask for the SSO administrator password and then proceed to regenerate and replace STS certificate.
    1. For example:
    2. This works on external and embedded PSCs
    3. This script will do the following
    4. Regenerate STS certificate
    5. What is needed?
    6. Offline snapshots of VCs/PSCs
    7.  SSO Admin Password

       

    8. IMPORTANT: This script should only be run on a single PSC per SSO domain
==================================
Resetting STS certificate for xxx.domain.local started on Fri May 22 14:39:40 UTC 2020

Detected DN: cn=xxxxxx ,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: xxxxxxx
Detected PSC: xxxxxxx
Detected SSO domain name: vsphere.local
Detected Machine ID: #######-####-####-#########3
Detected IP Address: 192.x.x.x
Domain CN: dc=vsphere,dc=local
==================================
==================================

Detected Root's certificate expiration date: 2030 May 16
Detected today's date: 2020 May 22
==================================

Exporting and generating STS certificate

Status : Success
Using config file : /tmp/vmware-fixsts/certool.cfg
Status : Success

Enter password for [email protected]:
Amount of tenant credentials: 1
Exporting tenant and trustedcertchain 1 to /tmp/vmware-fixsts

Deleting tenant and trustedcertchain 1

Applying newly generated STS certificate to SSO domain
adding new entry "cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"

adding new entry "cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"

Replacement finished - Please restart services on all vCenters and PSCs in your SSO domain
==================================
IMPORTANT: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
==================================
==================================

      Note: If you receive the following error when trying to run the script:
      bash: ./fixsts.sh: /bin/bash^M: bad interpreter: No such file or directory

  9. This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor. To resolve this problem, run this command and rerun the script:
     
    # sed -i -e 's/$//' fixsts.sh

Additional Information

Note:
Running the script will delete all the old STS chains and replace with a single new chain, resolving the authorization errors caused by the obsolete chains.
If there are multiple certificate chains and no LEAF in the (trustedcertchain) please proceed to regenerate the STS Certificate.

Powered by Wolken Software