Logging into vCenter fails with error | HTTP Status 500

Products

VMware vCenter Server VMware vCenter Server 7.0

Issue/Introduction

HTTP Status 500 - Internal Server Error while logging into vCenter from UI:

All services will be up and running.

Environment

VMware vCenter Server 7.0.x

Cause

This issue happens when the vsphere-webclient user is missing from ActAsUsers:

/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log:

[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-5090-exec-1         70000006 100004 ###### com.vmware.identity.token.impl.SamlTokenImpl                      SAML token for SubjectNameId [value=vsphere-webclient-****32ba-**08-46ee-****-d47504d9****@vsphere.local, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-5090-exec-1         70000006 100004 ###### com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl           Successfully acquired token for user: {Name: vsphere-webclient-***b32ba-ee08-**ee-****-d475**d9a11b, Domain: vsphere.local}
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-5090-exec-1         70000006 100004 ###### com.vmware.vise.vim.security.sso.impl.NgcSolutionUser             Solution user logged into domain vsphere.local
[YYYY-MM-DDTHH:MM:SS] [ERROR] http-nio-5090-exec-1         70000006 100004 ###### com.vmware.vise.vim.vapi.SessionBasedVapiConnectionAuthenticator Error while logging in to service 'https://vcenter_fqdn.local:443/site/api' - invoking 'create' on type 'com.vmware.cis.Session' failed. java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Caused by: com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vapi.endpoint.method.authentication.required,
    defaultMessage = Authentication required.,
    args = [],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = UNAUTHENTICATED,
    challenge = Basic realm="VAPI endpoint",SIGN realm=#################################,service="VAPI endpoint",sts=https://vcenter_fqdn/sts/STSService/vsphere.local
}
        at java.lang.Thread.getStackTrace(Thread.java:1559)
        at com.vmware.vapi.bindings.client.AsyncCallbackSyncAdapter.get_aroundBody1$advice(AsyncCallbackSyncAdapter.java:49)
        at com.vmware.vapi.bindings.client.AsyncCallbackSyncAdapter.get(AsyncCallbackSyncAdapter.java:1)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:145)
        at com.vmware.cis.SessionStub.create(SessionStub.java:46)
        at com.vmware.cis.SessionStub.create(SessionStub.java:37)
        ... 209 common frames omitted
Caused by: com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vapi.endpoint.method.authentication.required,
    defaultMessage = Authentication required.,
    args = [],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = UNAUTHENTICATED,
    challenge = Basic realm="VAPI endpoint",SIGN realm=#################################,service="VAPI endpoint",sts=https://vcenterfqdn.local/sts/STSService/vsphere.local
}
Caused by: com.vmware.vcenter.apigw.api.sso.SsoServiceException: Failed to acquire a token by token for domain vsphere.local(-****32ba-**08-46ee-****-d47504d9****)from STS http://localhost:1080/external-vecs/http2/vcenter_fqdn.local/443/sts/STSService/vsphere.local
        at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.acquireTokenByToken(SsoServiceImpl.java:663)
      at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.acquireDelegatedToken(SsoServiceImpl.java:471)
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.acquireDelegatedTokenFromSso(AsyncTokenProvider.java:1389)
        ... 8 common frames omitted
Caused by: com.vmware.vim.sso.client.exception.InvalidTokenRequestException: Request is invalid: ns0:InvalidRequest: Access not authorized!

Resolution

To resolve this issue, SSH to the affected node and identify the machine ID :

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Now check if the vsphere-webclient user is missing from ActAsUsers using the ldapsearch command.
In this example, the SSO domain is vsphere.local. If this has been changed from the default, alter the command to the correct SSO domain name. Include the SSO admin password in place of the hashes after the -w tag:

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=ActAsUsers,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w '########' member

For example:

root@vcenter [ ~ ]# ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=ActAsUsers,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w '########' member
dn: CN=ActAsUsers,DC=vsphere,DC=local
member: CN=vsphere-webclient-#################################,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=vpxd-#################################,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=vpxd-#################################,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=wcp-#################################,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=workload_storage_management-#################################,cn=ServicePrincipals,dc=vsphere,dc=local
member: CN=vpxd-#################################,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=vpxd-extension-#################################,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=wcp-#################################,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=workload_storage_management-#################################,cn=ServicePrincipals,dc=vsphere,dc=local

Here we have 2 vCenter's in linked mode and vsphere-webclient-################################# is missing as per the above output. The correct ID will be displayed in place of ################################# as seen in the above and below examples.

To fix this issue we need to add vsphere-webclient details to ActAsUsers.

Option 1:

cd /var/tmp
vi actas.ldif

dn: CN=ActAsUsers,DC=vsphere,DC=local
changetype: modify
add: member
member: CN=vsphere-webclient-#################################,CN=ServicePrincipals,DC=vsphere,DC=local

Make changes as per the environment and save the file.

Run:

/opt/likewise/bin/ldapadd -h localhost -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'VMware123!' -f actas.ldif

Expected output:

root@vcenter [ /var/tmp ]# /opt/likewise/bin/ldapadd -h localhost -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w '########' -f actas.ldif
modifying entry "CN=ActAsUsers,DC=vsphere,DC=local"

Option 2: Create a python script with the contents below

First, navigate to an appropriate directory and launch the vi editor using the following commands:

cd /var/tmp
vi actas.py

Then add the following to the new document:

import sys
import os
sys.path.append(os.environ['VMWARE_PYTHON_PATH'])
from cis.cisreglib import VmafdClient
from cis.vecs import SsoGroup
vmafd_client = VmafdClient()
machine_name = vmafd_client.get_machine_name()
machine_password = vmafd_client.get_machine_password()
machine_id = vmafd_client.get_machine_id()
solution_user_id = 'vsphere-webclient-' + machine_id
sso_group = SsoGroup(login=machine_name, password=machine_password)
sso_group.add_user('ActAsUsers', solution_user_id)

Copy paste above and save the file. Run !wq in vi to write and quit back to CLI. Then run the created python script with this command:

python actas.py

Restart the vsphere-ui service by running:

service-control --restart vsphere-ui

And try logging into the vCenter UI.