Post vCenter snapshot revert and reboot fails to bring up the svcs and wcp services.

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

• vCenter was reverted to snapshot after a  failed patch update and then was rebooted.
• After reboot, the vCenter services vpxd-svcs and wcp services would not start.
• In the vCenter /var/log/vmware/vmon/vmon.log we see similar error to below
  yyyy-mm-ddThh:mm:ss.829Z Wa(03) host-1887 <sps> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:22000/sms/HealthStatus. Exception: HTTP Error 403: AuthorizedOnly
yyyy-mm-ddThh:mm:ss.829Z Wa(03)+ host-1887
yyyy-mm-ddThh:mm:ssZ In(05) host-1887 <sps> Re-check service health since it is still initializing.
yyyy-mm-ddThh:mm:ssZ In(05) host-1887 <vc-ws1a-broker> Running the API Health command as user
yyyy-mm-ddThh:mm:ssZ In(05) host-1887 <vc-ws1a-broker-healthcmd> Constructed command: /storage/containers/vc-ws1a-broker/f6b616b765efe15183f0e49383402ac4cddf90c9cb649a17192637ba38825517/health_check.sh
yyyy-mm-ddThh:mm:ssZ Wa(03) host-1887 <vc-ws1a-broker> Service api healthcheck command returned unknown exit code 1
yyyy-mm-ddThh:mm:ssZ In(05) host-1887 <vc-ws1a-broker> Re-check service health since it is still initializing
yyyy-mm-ddThh:mm:ss.Z [Thread-24 [] ERROR com.vmware.sync.GrpcUtils  opId=] *** shutting down gRPC server since JVM is shutting down
yyyy-mm-ddThh:mm:ss.Z [Thread-29 [] INFO  com.vmware.vim.dataservices.VpxdSvcsMain  opId=] Shutting down the server
yyyy-mm-ddThh:mm:ss.Z [Thread-29 [] INFO  com.vmware.vim.dataservices.DataService  opId=] Stopping VLSI server...
yyyy-mm-ddThh:mm:ss.Z [Thread-29 [] INFO  com.vmware.vim.vmomi.core.common.impl.BasicLifecycleManager  opId=] Stopping lifecycle listeners.
yyyy-mm-ddThh:mm:ss.Z [Thread-24 [] ERROR com.vmware.sync.GrpcUtils  opId=] *** server shut down
yyyy-mm-ddThh:mm:ss.Z [Thread-29 [] INFO  com.vmware.vim.vmomi.server.http.impl.TcServer  opId=] Stopping server.
yyyy-mm-ddThh:mm:ss.Z [Thread-29 [] INFO  com.vmware.vim.dataservices.DataService  opId=] Shutting down caches...
yyyy-mm-ddThh:mm:ss.Z [Thread-29 [] INFO  com.vmware.vim.dataservices.DataService  opId=] Stopping security cache...
yyyy-mm-ddThh:mm:ss.Z [Thread-29 [] INFO  com.vmware.vim.dataservices.DataService  opId=] Stopping query service.
  
  
• Starting the vpxd-svcs service fails with below message

    service-control --start vpxd-svcs

    Operation not cancellable. Please wait for it to finish...
    Performing start operation on service vpxd-svcs...
    Error executing start on service vpxd-svcs. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
         }
     Service-control failed. Error: {
     "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null

 

• If you run wget http://localhost:22000/sms/HealthStatus it shows that it successfully connects to the configured proxy server. However it also reports error "Exception: HTTP Error 403: AuthorizedOnly" as it also tries to connect to local proxy server.

 

Resolution

To resolve this issue  add the entry localhost,127.0.0.1 in the proxy configuration file under NO_PROXY as below

1. Open an SSH connection to the vCenter Server Appliance and log in as root

2. Navigate to the directory /etc/sysconfig/proxy using command

     # cd /etc/sysconfig
 

3. Open the proxy configuration file "proxy" using the VI editor:

# vi proxy
 

4. To exclude traffic to specific systems from being routed via the proxy, edit the NO_PROXY option to include both the FQDN and IP address of the system. Be aware though, that NO_PROXY requires specific syntax. To separate entries in this option, use a comma (,) followed by a space ( ):

# Example: NO_PROXY="www.example.com, example.com, localhost"
NO_PROXY="localhost, 127.0.0.1, vc.example.com, 192.168.1.55"

Ensure to add localhost, 127.0.0.1.

Save the changes using wq! 

5. Reboot the VC VM  and check if the services are started.