After migrating vCenter Server 6.0 from an Embedded Platform Services Controller to External Platform Services Controller the Performance Chart Service fails with the error: Unable to Initialize servlet
search cancel

After migrating vCenter Server 6.0 from an Embedded Platform Services Controller to External Platform Services Controller the Performance Chart Service fails with the error: Unable to Initialize servlet

book

Article ID: 344694

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
After migrating vCenter Server 6.0 from an Embedded Platform Services Controller (PSC) to External Platform Services Controller, you experience these symptoms:

  • Under the System Configuration UI (Administration > System Configuration), the VMware Performance Chart Service reports this error:

    Unable to Initialize servlet
    Failed to request health status from URI <FQDN of Embedded Platform Services Controller>

     
  • Under the Alarms pane, you see the Performance Charts Service Health Alarm in a Critical status.
  • In the /var/log/vmware/perfcharts or %ProgramData%\VMware\vCenterServer\logs\perfcharts stats.log file, you see entries similar to:

    YYYY-MM-DDT<time>Z [pool-1-thread-1 ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl] Error communicating to the remote server https://embedded_psc.vmware.com/sts/STSService/vsphere.local
    com.sun.xml.internal.ws.client.ClientTransportException: The server sent HTTP status code 503: Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7f014c066a10] _serverNamespace = /sts/STSService _isRedirect = false _port = 7080)
    at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.checkStatusCode(Unknown Source)
    ...
    YYYY-MM-DDT<time>Z [pool-1-thread-1 ERROR com.vmware.vim.stats.webui.util.ResourceModelClient] Error communicating to the remote server https://embedded_psc.vmware.com/sts/STSService/vsphere.local com.vmware.vim.sso.client.exception.ServerCommunicationException: Error communicating to the remote server https://blr7-7th-dhcp-44-37.eng.vmware.com/sts/STSService/vsphere.local
    at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:781)
    at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:699)
    at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:460)
    at com.vmware.vim.stats.webui.util.ResourceModelClient.exchangeForActAsToken(Unknown Source)
    at com.vmware.vim.stats.webui.util.ResourceModelClient.<init>(Unknown Source)
    ...
    YYYY-MM-DDT<time>Z [pool-1-thread-1 WARN com.vmware.vim.stats.webui.startup.StatsReportInitializer] STATs report initialization failed. Set health status to RED.
    </time></time></time>

    Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.



Environment

VMware vCenter Server 6.0.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 6.5.x

Cause

This issue occurs when the VMware Performance Chart Service of vCenter Server attempts to connect to the decommissioned Embedded PSC's Secure Token Service (STS) to perform a SAML exchange.

Resolution

To resolve this issue, un-register the old Secure Token Service service registration in the Lookup Service:
 
For the Platform Services Controller Appliance:
  1. Connect to the External Platform Service Controller Appliance with an SSH session.
  2. Provide the root user user name and password when prompted.
  3. Run this command to enable the Bash shell:

    shell.set --enable True
     
  4. Run this command to access the Bash shell:

    shell
     
  5. Run this command to navigate to the scripts directory:

    cd /usr/lib/vmidentity/tools/scripts
     
  6. Run this command to list the STS service registrations:

    ./lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://External_PSC_FQDN/lookupservice/sdk 2> /dev/null

    For example:

    ./lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://psc.example.com/lookupservice/sdk 2> /dev/null
     
  7. This should report a minimum of two endpoints. One belonging to the new, external PSC and one belonging to the decommissioned, Embedded PSC.

    For example:

    Service Product: com.vmware.cis
    Service Type: cs.identity
    Service ID: ########-####-####-####-########ac8a
    Site ID: site11
    Owner ID: [email protected]
    Version: 2.0
    Endpoints:
    Type: com.vmware.cis.cs.identity.admin
    Protocol: wsTrust
    URL: https://external_psc.example.com/sts/STSService/vsphere.local
    SSL trust: <SSL Certificate>

    --------------------------------------------------

    Service Product: com.vmware.cis
    Service Type: cs.identity
    Service ID: ########-####-####-####-########fb44
    Site ID: site11
    Owner ID: [email protected]
    Version: 2.0
    Endpoints:
    Type: com.vmware.cis.cs.identity.admin
    Protocol: wsTrust
    URL: https://embedded_psc.example.com/sts/STSService/vsphere.local
    SSL trust: <SSL Certificate>

     
  8. Run this command to un-register the old STS service registration:

    ./lstool.py unregister --user "[email protected]" --password "administrator password" --id <Embedded PSC Service ID From above> --no-check-cert --url https://External_PSC_FQDN/lookupservice/sdk

    For example:

    ./lstool.py unregister --user [email protected] --password --id ########-####-####-####-########fb44 --no-check-cert --url https://psc.example.com/lookupservice/sdk
     
  9. On the affected vCenter Server, restart the VMware Performance Chart Service by running these commands:

    service-control --stop vmware-perfcharts

    service-control --start vmware-perfcharts
 
For the Platform Services Controller for Windows:
  1. Remote Desktop into the Windows Server.
  2. Open an elevated command prompt.
  3. Run this command to navigate to the scripts directory:

    cd "C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts"
     
  4. Run this command to list the STS service registrations:

    "%VMWARE_PYTHON_BIN%" lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://External_PSC_FQDN/lookupservice/sdk

    For example:

    "%VMWARE_PYTHON_BIN%" lstool.py list --ep-type com.vmware.cis.cs.identity.sso --no-check-cert --url https://psc.example.com/lookupservice/sdk
     
  5. This should report a minimum of two endpoints. One belonging to the new, external PSC and one belonging to the decommissioned, Embedded PSC.

    For example:

    Service Product: com.vmware.cis
    Service Type: cs.identity
    Service ID: ########-####-####-####-########ac8a
    Site ID: site11
    Owner ID: [email protected]
    Version: 2.0
    Endpoints:
    Type: com.vmware.cis.cs.identity.admin
    Protocol: wsTrust
    URL: https://external_psc.example.com/sts/STSService/vsphere.local
    SSL trust: <SSL Certificate>

    --------------------------------------------------

    Service Product: com.vmware.cis
    Service Type: cs.identity
    Service ID: ########-####-####-####-########fb44
    Site ID: site11
    Owner ID: [email protected]
    Version: 2.0
    Endpoints:
    Type: com.vmware.cis.cs.identity.admin
    Protocol: wsTrust
    URL: https://embedded_psc.example.com/sts/STSService/vsphere.local
    SSL trust: <SSL Certificate>

     
  6. Run this command to un-register the old STS service registration:

    "%VMWARE_PYTHON_BIN%" lstool.py unregister --user "[email protected]" --password "administrator password" --id <Embedded PSC Service ID From above> --no-check-cert --url https://External_PSC_FQDN/lookupservice/sdk

    For example:

    "%VMWARE_PYTHON_BIN%" lstool.py unregister --user [email protected] --password --id ########-####-####-####-########fb44 --no-check-cert --url https://psc.example.com/lookupservice/sdk
     
  7. On the affected vCenter Server, restart the VMware Performance Chart Service by running these commands:

    service-control --stop vmware-perfcharts

    service-control --start vmware-perfcharts



Additional Information

\