After migrating vCenter Server 6.0 from an Embedded Platform Services Controller to External Platform Services Controller the Performance Chart Service fails with the error: Unable to Initialize servlet

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
After migrating vCenter Server 6.0 from an Embedded Platform Services Controller (PSC) to External Platform Services Controller, you experience these symptoms:

Under the System Configuration UI (Administration > System Configuration), the VMware Performance Chart Service reports this error:

Unable to Initialize servlet
Failed to request health status from URI <FQDN of Embedded Platform Services Controller>
Under the Alarms pane, you see the Performance Charts Service Health Alarm in a Critical status.
In the /var/log/vmware/perfcharts or %ProgramData%\VMware\vCenterServer\logs\perfcharts stats.log file, you see entries similar to:

YYYY-MM-DDT<time>Z [pool-1-thread-1 ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl] Error communicating to the remote server https://embedded_psc.vmware.com/sts/STSService/vsphere.local
com.sun.xml.internal.ws.client.ClientTransportException: The server sent HTTP status code 503: Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7f014c066a10] _serverNamespace = /sts/STSService _isRedirect = false _port = 7080)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.checkStatusCode(Unknown Source)
...
YYYY-MM-DDT<time>Z [pool-1-thread-1 ERROR com.vmware.vim.stats.webui.util.ResourceModelClient] Error communicating to the remote server https://embedded_psc.vmware.com/sts/STSService/vsphere.local com.vmware.vim.sso.client.exception.ServerCommunicationException: Error communicating to the remote server https://blr7-7th-dhcp-44-37.eng.vmware.com/sts/STSService/vsphere.local
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:781)
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:699)
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:460)
at com.vmware.vim.stats.webui.util.ResourceModelClient.exchangeForActAsToken(Unknown Source)
at com.vmware.vim.stats.webui.util.ResourceModelClient.<init>(Unknown Source)
...
YYYY-MM-DDT<time>Z [pool-1-thread-1 WARN com.vmware.vim.stats.webui.startup.StatsReportInitializer] STATs report initialization failed. Set health status to RED.
</time></time></time>
Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.0.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 6.5.x

Cause

This issue occurs when the VMware Performance Chart Service of vCenter Server attempts to connect to the decommissioned Embedded PSC's Secure Token Service (STS) to perform a SAML exchange.

Resolution

To resolve this issue, un-register the old Secure Token Service service registration in the Lookup Service:

For the Platform Services Controller Appliance:

Connect to the External Platform Service Controller Appliance with an SSH session.
Provide the root user user name and password when prompted.
Run this command to enable the Bash shell:

shell.set --enable True
Run this command to access the Bash shell:

shell
Run this command to navigate to the scripts directory:

cd /usr/lib/vmidentity/tools/scripts
Run this command to list the STS service registrations:

./lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://External_PSC_FQDN/lookupservice/sdk 2> /dev/null

For example:

./lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://psc.example.com/lookupservice/sdk 2> /dev/null
This should report a minimum of two endpoints. One belonging to the new, external PSC and one belonging to the decommissioned, Embedded PSC.

For example:

Service Product: com.vmware.cis
Service Type: cs.identity
Service ID: ########-####-####-####-########ac8a
Site ID: site11
Owner ID: [email protected]
Version: 2.0
Endpoints:
Type: com.vmware.cis.cs.identity.admin
Protocol: wsTrust
URL: https://external_psc.example.com/sts/STSService/vsphere.local
SSL trust: <SSL Certificate>

--------------------------------------------------

Service Product: com.vmware.cis
Service Type: cs.identity
Service ID: ########-####-####-####-########fb44
Site ID: site11
Owner ID: [email protected]
Version: 2.0
Endpoints:
Type: com.vmware.cis.cs.identity.admin
Protocol: wsTrust
URL: https://embedded_psc.example.com/sts/STSService/vsphere.local
SSL trust: <SSL Certificate>
Run this command to un-register the old STS service registration:

./lstool.py unregister --user "[email protected]" --password "administrator password" --id <Embedded PSC Service ID From above> --no-check-cert --url https://External_PSC_FQDN/lookupservice/sdk

For example:

./lstool.py unregister --user [email protected] --password --id ########-####-####-####-########fb44 --no-check-cert --url https://psc.example.com/lookupservice/sdk
On the affected vCenter Server, restart the VMware Performance Chart Service by running these commands:

service-control --stop vmware-perfcharts

service-control --start vmware-perfcharts

For the Platform Services Controller for Windows:

Remote Desktop into the Windows Server.
Open an elevated command prompt.
Run this command to navigate to the scripts directory:

cd "C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts"
Run this command to list the STS service registrations:

"%VMWARE_PYTHON_BIN%" lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://External_PSC_FQDN/lookupservice/sdk

For example:

"%VMWARE_PYTHON_BIN%" lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://psc.example.com/lookupservice/sdk
This should report a minimum of two endpoints. One belonging to the new, external PSC and one belonging to the decommissioned, Embedded PSC.

For example:

Service Product: com.vmware.cis
Service Type: cs.identity
Service ID: ########-####-####-####-########ac8a
Site ID: site11
Owner ID: [email protected]
Version: 2.0
Endpoints:
Type: com.vmware.cis.cs.identity.admin
Protocol: wsTrust
URL: https://external_psc.example.com/sts/STSService/vsphere.local
SSL trust: <SSL Certificate>

--------------------------------------------------

Service Product: com.vmware.cis
Service Type: cs.identity
Service ID: ########-####-####-####-########fb44
Site ID: site11
Owner ID: [email protected]
Version: 2.0
Endpoints:
Type: com.vmware.cis.cs.identity.admin
Protocol: wsTrust
URL: https://embedded_psc.example.com/sts/STSService/vsphere.local
SSL trust: <SSL Certificate>
Run this command to un-register the old STS service registration:

"%VMWARE_PYTHON_BIN%" lstool.py unregister --user "[email protected]" --password "administrator password" --id <Embedded PSC Service ID From above> --no-check-cert --url https://External_PSC_FQDN/lookupservice/sdk

For example:

"%VMWARE_PYTHON_BIN%" lstool.py unregister --user [email protected] --password --id ########-####-####-####-########fb44 --no-check-cert --url https://psc.example.com/lookupservice/sdk
On the affected vCenter Server, restart the VMware Performance Chart Service by running these commands:

service-control --stop vmware-perfcharts

service-control --start vmware-perfcharts

Additional Information

\