Slow or invalid search results in vSphere UI with domain accounts in vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

• When logged in with an Active Domain user account, searching using the vCenter UI the search bar result spins in a loop with the "Searching..." message but does not provide any results
  
• When using [email protected] the vCenter UI completes the search without issue
• Logging in with an AD user account can take a long time to complete too 
• This can occur cluster wide in vCenter ELM clusters

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

The vCenter search bar is unresponsive due to communication errors caused by IWA (Integrated Windows Authentication) identity source set up with multiple domain controllers either having latency or connection issues with the vCenter, or being completely unavailable to the vCenter.

Some vCenter logs will indicate connection issues as seen below:

• /var/log/vmware/vsphere-ui/logs/access/localhost_access.log:
  

  127.0.0.1 ###.###.##.## - - [YYYY-MM-DDTHH:MM:SS] "GET /ui/search/quicksearch/?opId=:1769292260&query=051p HTTP/1.1" 200 231 200022 70002088 http-nio-127.0.0.1-5090-exec-42 59307
  127.0.0.1 ###.###.##.## - - [YYYY-MM-DDTHH:MM:SS] "GET /ui/search/quicksearch/?opId=:1751538719&query=051 HTTP/1.1" 200 643 200022 70002083 http-nio-127.0.0.1-5090-exec-37 59899
  127.0.0.1 ###.###.##.## - - [YYYY-MM-DDTHH:MM:SS] "GET /ui/search/quicksearch/?opId=:1414305137&query=0 HTTP/1.1" 200 1491 200022 70002080 http-nio-127.0.0.1-5090-exec-11 60409

• /var/log/vmware/vsphere-ui/logs/dataservice.log:


  [YYYY-MM-DDTHH:MM:SS.000Z] [ERROR] data-service-pool-756        70011266 100296 200067 c.v.v.client.folder.LazyLoadingDataMigrationPropertyProvider      Exception is thrown while retrieving lazyImportData propertiesjava.lang.InterruptedException
  [YYYY-MM-DDTHH:MM:SS.000Z] [ERROR] data-service-pool-762        70011316 100299 200067 c.v.v.client.folder.LazyLoadingDataMigrationPropertyProvider      Exception is thrown while retrieving lazyImportData propertiesjava.lang.InterruptedException
  [YYYY-MM-DDTHH:MM:SS.000Z] [ERROR] data-service-pool-753        70011471 100280 200067 c.vmware.vsphere.client.folder.impl.VCenterKmipPropertyProvider   Failed to retrieve getDefaultKmsCluster java.lang.NullPointerException: null
  [YYYY-MM-DDTHH:MM:SS.000Z] [ERROR] -nio-127.0.0.1-5090-exec-121 70011495 100267 200065 com.vmware.vise.mvc.exception.GlobalExceptionHandler              Exception handled while processing request for /ui/data/propertiesByRelation/urn:vmomi:Folder:group-d1:5118bf09-fcb0-4f6e-8892-f
  630d7a098f9?properties=name,description,categoryName&relation=entity&targetType=inventoryservice:InventoryServiceTag:  com.vmware.vise.data.query.internal.DataServiceException: The query execution timed out because of a back-end data adapter 'com.vmware.vise.data.adapters.core.DataServiceCoreAdapter' which took more than 120 seconds.
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:83)
  [YYYY-MM-DDTHH:MM:SS.000Z] [ERROR] data-service-pool-768        70011499 100267 200065 com.vmware.vsphere.client.h5.vcha.VchaPropertyProviderAdapter     Exception thrown while retrieving vcha propertiesjava.lang.InterruptedException
  

Resolution

• IWA has been depreciated as of vCenter 7.x. Refer to KB: Removal of Integrated Windows Authentication (IWA)
• Change the vCenter identity source from IWA to Active Directory over LDAP (or LDAPs) and define specific domain controllers accessible to the vCenter in order to resolve the search timeout issue.