vSAN skyline health is not visible on vCenter UI, post upgrading the vCenter to 8.0.3

Products

• VMware vSAN 8.x
• VMware vCenter Server 8.0

Issue/Introduction

• After upgrading the vCenter to 8.0 U3 in GUI, the vSAN menu does not show up on monitor/configure tab.
• Errors are seen in /var/log/vmware/vsphere-ui/logs/vsphere_client.log
  

  #################[ERROR] plugin-discovery-bus-thread  70624012 107965 200768 com.vmware.vise.plugin.proxyconfig.ReverseProxyConfigurationApi   An error occurred while creating/updating reverse proxy configuration file remote-plugins.json. java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
      messages = [],
      data = <null>,
      errorType = ERROR
  }
          at java.util.concurrent.FutureTask.report(FutureTask.java:122)
          at java.util.concurrent.FutureTask.get(FutureTask.java:206)
          at com.vmware.vise.plugin.proxyconfig.ReverseProxyConfigurationApi.configureProxy(ReverseProxyConfigurationApi.java:658)
          at com.vmware.vise.plugin.proxyconfig.ReverseProxyConfigurationApi.addRoutingRules(ReverseProxyConfigurationApi.java:182)
          at com.vmware.vise.plugin.proxyconfig.JsonFileBasedRemotePluginReverseProxyConfigurer.addProxyRules(JsonFileBasedRemotePluginReverseProxyConfigurer.java:96)
          at com.vmware.vise.plugin.proxyconfig.RemotePluginReverseProxyServiceImpl.addProxyRules(RemotePluginReverseProxyServiceImpl.java:83)
          at com.vmware.vise.plugin.extension.VcExtensionManager.processRemotePlugin(VcExtensionManager.java:1332)
          at com.vmware.vise.plugin.extension.VcExtensionManager.addPackages(VcExtensionManager.java:1289)
          at com.vmware.vise.plugin.extension.VcExtensionManager.retryFailedPackages(VcExtensionManager.java:1193)
          at com.vmware.vise.plugin.extension.CorePluginsDeployer.retryFailedPackages(CorePluginsDeployer.java:110)
          at com.vmware.vise.plugin.discovery.impl.PluginDiscoveryAgent.incrementalDeployCorePackages(PluginDiscoveryAgent.java:139)
          at com.vmware.vise.plugin.discovery.impl.PluginDiscoveryAgent.onPluginDiscoveryOnLoginRequest(PluginDiscoveryAgent.java:86)
          at sun.reflect.GeneratedMethodAccessor4454.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
          at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:142)
          at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:750)
  Caused by: com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
      messages = [],
      data = <null>,
      errorType = ERROR
  ################# [WARN ] plugin-discovery-bus-thread  70624012 107965 200768 com.vmware.bifrost.bus.EventBusLowApiImpl                         Failed to send message. Cannot find channel: plugin-state-change-notification
  ################# [ERROR] plugin-discovery-bus-thread  70624012 107965 200768 com.vmware.vise.plugin.status.RemotePluginStatusServiceImpl       DEPLOYMENT_FAILED: Error deploying plugin package com.vmware.vsan.client:8.0.203.10000. Reason: Plugin configuration with Reverse Proxy failed.
  ################# [INFO ] plugin-discovery-bus-thread  70624012 107965 200768 com.vmware.vise.plugin.registry.VcExtensionStateRegistry          Updating entry: Plugin: 'com.vmware.vsan.client:8.0.203.10000', State: 'FAILED_CONFIGURE'

• /var/log/vmware/rhttproxy/rhttproxy.log 

  ################# error rhttpproxy[02735] [Originator@6876 sub=RhttpProxy] [Rhttpproxy JWT] JWT token has expired!
  #################  error rhttpproxy[02735] [Originator@6876 sub=RhttpProxy] [Rhttpproxy REST PUT Handler] JWT verification failed
  #################  info rhttpproxy[02738] [Originator@6876 sub=RhttpProxy.HTTPService.HttpConnection] Failed to read header; <io_obj p:0x00007f15a40ecaa0, h:21, <TCP '127.0.0.1 : 549'>, <TCP '127.0.0.1 : 35198'>>: N7Vmacore21NotSupportedExceptionE(Not Supported)

 

Environment

VMware vCenter Server 8.0 U3

Cause

• The vSAN plugin download is failing due to reverse proxy configuration failed. The rhttpproxy have recently enabled the JWT authentication for their REST APIs. The vSphere-UI code was assuming that the JWT is needed only for requests to the new REST API but this is incorrect - it's also needed for the old file-based APIs.
  
  

Resolution

1. Upgrade the vCenter version to 8.0U3b(P04) 24262322