STS Health Status reports 'Health status changed alarm' or 'sts status changed from green to red'

Products

• VMware vCenter Server

Issue/Introduction

• vCenter reports an alarm:
  Alarm 'Health status changed alarm' on Datacenters triggered by event #### 'sts status changed from green to red]
  
  
• Restarting vCenter services via service-control commands temporarily resolves the issue
  
  
• The sts-health-status.log shows health status as RED:
  [YYYY-MM-DDTHH:MM:SS] ERROR - Publishing health status as RED to vMon.
  

• The /var/log/vmware/sso/vmware-identity-sts-default.log shows 'failed to get attributes for principal':
[YYYY-MM-DDTHH:MM:SS] ERROR sts-default[30:Thread-9][com.vmware.identity.idm.server.IdentityManager] Failed to get attributes for principal [machine-####-####-####-####-####@vsphere.local] in tenant [vsphere.local]
[YYYY-MM-DDTHH:MM:SS] ERROR sts-default[30:Thread-9][com.vmware.identity.idm.server.ServerUtils] Exception 'java.lang.NullPointerException'
java.lang.NullPointerException: null
        at com.vmware.identity.idm.server.IdentityManager.getForeignTrustProviderIfExternalIdp(IdentityManager.java:13076) ~[libvmware-identity-idm-server.jar:?]
        at com.vmware.identity.idm.server.IdentityManager.getAttributeValues(IdentityManager.java:3910) ~[libvmware-identity-idm-server.jar:?]
        at com.vmware.identity.idm.server.IdentityManager.getAttributeValues(IdentityManager.java:10689) [libvmware-identity-idm-server.jar:?]
        at com.vmware.identity.idm.client.CasIdmClient.getAttributeValues(CasIdmClient.java:1420) [libvmware-identity-idm-client.jar:?]
        at com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor.getAttributes(IdmPrincipalAttributesExtractor.java:119) [libsamlauthority.jar:?]
        at com.vmware.identity.saml.impl.TokenAuthorityImpl.getTokenAttributesAndIdentityAttribute(TokenAuthorityImpl.java:332) [libsamlauthority.jar:?]
        at com.vmware.identity.saml.impl.TokenAuthorityImpl.createAssertion(TokenAuthorityImpl.java:283) [libsamlauthority.jar:?]
        at com.vmware.identity.saml.impl.TokenAuthorityImpl.issueToken(TokenAuthorityImpl.java:190) [libsamlauthority.jar:?]
        at com.vmware.identity.providers.SolutionUserHokTokenProviderImpl.getToken(SolutionUserHokTokenProviderImpl.java:65) [libsamlauthority.jar:?]
        at com.vmware.identity.util.VapiClientConnection.createConnection(VapiClientConnection.java:88) [libsamlauthority.jar:?]
        at com.vmware.identity.util.VapiClientConnection.refreshConnection(VapiClientConnection.java:157) [libsamlauthority.jar:?]
        at com.vmware.identity.util.VapiClientConnection.invokeStub(VapiClientConnection.java:272) [libsamlauthority.jar:?]
        at com.vmware.identity.util.VapiClient.doVcTrustsList(VapiClient.java:51) [libsamlauthority.jar:?]
        at com.vmware.identity.util.VcTrustCache.refreshTrustCache(VcTrustCache.java:423) [libsamlauthority.jar:?]
        at com.vmware.identity.util.VcTrustCache$TrustCacheThread.run(VcTrustCache.java:468) [libsamlauthority.jar:?]

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vCenter Server 9.0.x

Cause

The Machine GUID is missing from the vCenter server vmdir database.

Resolution

1. Take an offline (powered off) snapshot of all affected vCenter servers
   
   
2. Validate the existence of MachineGuid. SSH into the vCenter server, then:
    /opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\vmdir]' | grep MachineGuid
   Take note of the MachineGuid.
   
   
3. Add/Recreate the MachineGuid:
    /opt/likewise/bin/lwregshell add_value '[HKEY_THIS_MACHINE\Services\vmdir]' MachineGuid REG_SZ 1
 /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "MachineGuid" "####-####-####-####-####" 
   Replace "####-####-####-####-####" with the MachineGuid found in Step 2.
   
   
4. Restart all vCenter services:
    service-control --stop --all && service-control --start --all