In the lookupserver.log file, you see entries similar to:
inherited from com.vmware.vim.binding.lookup.SearchCriteria@780d5011) because of Invalid certificate java.lang.IllegalArgumentException: Invalid certificate
Environment
VMware vCenter Server 5.1.x
Cause
This issue occurs when a service endpoint has an expired trust while attempting to register with the Lookup Service.
Resolution
This is a known issue affecting VMware vCenter Server 5.1.x.
Note: Before proceeding to step 21, take a backup of the folder, located at c:\programdata\vmware\virtual center\ssl and replace the contents with the new certificate. You cannot register vCenter Server with Single Sign-On if the current certificates are expired.