Unable to access vCenter, vpxd service not starting after configuring ADFS authentication through SDDC Manager in a VCF Environment

Products

• VMware Cloud Foundation
• VMware vCenter Server

Issue/Introduction

Symptoms:
After configuring ADFS Authentication from SDDC Manager UI in a VCF Environment, we will no longer be able to login to vCenter due to the vpxd service crashing, after a restart of the vCenter or vCenter services.

Accessing the vCenter UI shows an error of:

no healthy upstream

The vpxd.log in the vCenter server shows errors similar to:

YYYY-MM-DDTHH:MM:SS warning vpxd[40734] [Originator@6876 sub=vmomi.soapStub[11]] SOAP request returned HTTP failure; <<io_obj p:0x00007f1aa42c02f8, h:39, 
<TCP '127.0.0.1 : 55850'>, <TCP '127.0.0.1 : 10080'>>, /invsvc/vmomi/sdk>, method: loginBySamlToken; code: 500(Internal Server Error) 
YYYY-MM-DDTHH:MM:SS warning vpxd[40734] [Originator@6876 sub=Authz] [ConnectAndLogin] 
Failed to loginBySamlToken: N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.SecurityError
--> )
-->
YYYY-MM-DDTHH:MM:SS info vpxd[40734] [Originator@6876 sub=Authz] fallback to loginByCertificate
YYYY-MM-DDTHH:MM:SS error vpxd[40734] [Originator@6876 sub=ServerAccess] HTTP error while invoking remote login: 400 : Bad Request 
YYYY-MM-DDTHH:MM:SS error vpxd[40734] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr]
Failed to connect to Authz service: <N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication

Environment

VMware Cloud Foundation 4.5

Cause

This is due to an issue that has been identified in the workflow of configuring ADFS Identity Source specifically from the SDDC Manager UI in VCF 4.5.

Resolution

This is resolved in VCF 4.5.1 and above.

Workaround:
At a high level, the workaround steps involve:
- Unconfiguring the current ADFS Identity Source from vCenter
- Restarting all services or rebooting the vCenter to bring it back online
- Re-configuring ADFS Identity Source and Authentication at a vCenter level (If needed)

Workaround Steps:

1. In a browser tab, go specifically to the vCenter UI login URL:

https://<vc_fqdn>/ui/login

For example:

https://vcenter-1.example.com/ui/login

2. Login with [email protected] credentials.

3. Go to Administration > Single Sign On - Configuration > Identity Provider > Change Identity Provider



4. Select the Embedded Identity Provider.



5. Check the box for I understand the consequences and agree to continue configuring embedded identity provider and click Finish.



6. Reboot the vCenter or restart the services using the following command in an SSH session to the VC:

service-control --stop --all && service-control --start --all

7. Once the vCenter is back online, confirm that we are able to login to the vCenter and view its Inventory. 
At that point, if required, re-configure ADFS Authentication at a vCenter level.
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-C5E998B2-1148-46DC-990E-A5DB71F93351.html

Once the configuration is complete, SDDC Manager will reflect the ADFS Identity Source in the SDDC Manager UI as well.

Additional Information

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-C5E998B2-1148-46DC-990E-A5DB71F93351.html