After migrating vCenter Server 6.0 from an Embedded Platform Services Controller to External Platform Services Controller the Performance Chart Service fails with the error: Unable to Initialize servlet
book
Article ID: 344694
calendar_today
Updated On:
Products
VMware vCenter Server
Issue/Introduction
Symptoms: After migrating vCenter Server 6.0 from an Embedded Platform Services Controller (PSC) to External Platform Services Controller, you experience these symptoms:
Under the System Configuration UI (Administration > System Configuration), the VMware Performance Chart Service reports this error:
Unable to Initialize servlet Failed to request health status from URI <FQDN of Embedded Platform Services Controller>
Under the Alarms pane, you see the Performance Charts Service Health Alarm in a Critical status.
In the /var/log/vmware/perfcharts or %ProgramData%\VMware\vCenterServer\logs\perfcharts stats.log file, you see entries similar to:
YYYY-MM-DDT<time>Z [pool-1-thread-1 ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl] Error communicating to the remote server https://embedded_psc.vmware.com/sts/STSService/vsphere.local com.sun.xml.internal.ws.client.ClientTransportException: The server sent HTTP status code 503: Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7f014c066a10] _serverNamespace = /sts/STSService _isRedirect = false _port = 7080) at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.checkStatusCode(Unknown Source) ... YYYY-MM-DDT<time>Z [pool-1-thread-1 ERROR com.vmware.vim.stats.webui.util.ResourceModelClient] Error communicating to the remote server https://embedded_psc.vmware.com/sts/STSService/vsphere.local com.vmware.vim.sso.client.exception.ServerCommunicationException: Error communicating to the remote server https://blr7-7th-dhcp-44-37.eng.vmware.com/sts/STSService/vsphere.local at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:781) at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:699) at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:460) at com.vmware.vim.stats.webui.util.ResourceModelClient.exchangeForActAsToken(Unknown Source) at com.vmware.vim.stats.webui.util.ResourceModelClient.<init>(Unknown Source) ... YYYY-MM-DDT<time>Z [pool-1-thread-1 WARN com.vmware.vim.stats.webui.startup.StatsReportInitializer] STATs report initialization failed. Set health status to RED. </time></time></time> Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware vCenter Server 6.0.x VMware vCenter Server Appliance 6.5.x VMware vCenter Server 6.7.x VMware vCenter Server Appliance 6.0.x VMware vCenter Server Appliance 6.7.x VMware vCenter Server 6.5.x
Cause
This issue occurs when the VMware Performance Chart Service of vCenter Server attempts to connect to the decommissioned Embedded PSC's Secure Token Service (STS) to perform a SAML exchange.
Resolution
To resolve this issue, un-register the old Secure Token Service service registration in the Lookup Service:
For the Platform Services Controller Appliance:
Connect to the External Platform Service Controller Appliance with an SSH session.
Provide the root user user name and password when prompted.
Run this command to enable the Bash shell:
shell.set --enable True
Run this command to access the Bash shell:
shell
Run this command to navigate to the scripts directory:
cd /usr/lib/vmidentity/tools/scripts
Run this command to list the STS service registrations:
./lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://External_PSC_FQDN/lookupservice/sdk 2> /dev/null
For example:
./lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://psc.example.com/lookupservice/sdk 2> /dev/null
This should report a minimum of two endpoints. One belonging to the new, external PSC and one belonging to the decommissioned, Embedded PSC.
For example:
Service Product: com.vmware.cis Service Type: cs.identity Service ID: ########-####-####-####-########ac8a Site ID: site11 Owner ID: [email protected] Version: 2.0 Endpoints: Type: com.vmware.cis.cs.identity.admin Protocol: wsTrust URL: https://external_psc.example.com/sts/STSService/vsphere.local SSL trust: <SSL Certificate>