Multiple services on vCenter fail to start due to a failure to start the rhttpproxy service
search cancel

Multiple services on vCenter fail to start due to a failure to start the rhttpproxy service

book

Article ID: 320582

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

 

  • Multiple services fail to start on vCenter, with an error similar to the following: 
    Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start certificateauthority, sps, updatemgr, vpxd, wcp, vstats, hvc, topologysvc, sca, rhttpproxy, vpxd-svcs, certificatemanagement, content-library, analytics services. Error: Operation timed out
  • Multiple services fail to start on  error "Connection refused"
  • /var/log/vmware/analytics/analytics.log: 
    YYYY-MM-DDThh:mm:ss main  FATAL vmware.ph.phservice.service.Main Error starting service:  org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'retryOnInvalidSignatureVerificator' defined in class path resource [ph-vapi-server.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vapi.cis.authn.json.RetryJsonSignatureVerificationProcessor]: Constructor threw exception; nested exception is com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:1080/lookupservice/sdk invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
  • /var/log/vmware/sca/cmon.stderr:
     
    SEVERE: Failed to get STS issuers certificates; uri:http://localhost:1080/sso-adminserver/system-sdk/ [com.vmware.vim.sso.admin.exception.ConnectionException: http://localhost:1080/sso-adminserver/system-sdk/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"]
com.vmware.vim.sso.admin.exception.ConnectionException: http://localhost:1080/sso-adminserver/system-sdk/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
  • /var/log/vmware/vmware-sps/sps.log:
     
    YYYY-MM-DDThh:mm:ss [main] ERROR opId=sps-Main-535971-270 com.vmware.vim.storage.common.identity.ServiceAccountUserByHoKToken - Failed to login SPS:
com.vmware.vim.storage.common.serviceclient.identity.SsoException: Failed to create SSO admin client
...
Caused by: com.vmware.vim.sso.admin.exception.ConnectionException: http://localhost:1080/sso-adminserver/system-sdk invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"

  • /var/log/vmware/vapi/endpoint/endpoint.log:
     
    YYYY-MM-DDThh:mm:ss | ERROR | state-manager1            | DefaultStateManager            | Unexpected error while initializing endpoint runtime state.
com.vmware.vim.sso.admin.exception.ConnectionException: http://localhost:1080/sso-adminserver/system-sdk invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:1080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
  • /var/log/vmware/rhttpproxy/rhttpproxy.log file does not get updated.
  • var/log/vmware/rhttpproxy/rproxy_init.log.stderr 
     
    Starting service process with pid: 10301.
Application error: mismatched tag
Backtrace:
[backtrace begin] product: rhttpproxy, version: 8.0.1, build: build-21815093, tag: -, cpu: x86_64, os: linux, buildType: release
backtrace[00] rhttpproxy[0x005888B6]
backtrace[01] rhttpproxy[0x006FE5E1]
backtrace[02] rhttpproxy[0x006B9353]
backtrace[03] rhttpproxy[0x00767D2C]
backtrace[04] rhttpproxy[0x0077E91B]
backtrace[05] rhttpproxy[0x00765210]
backtrace[06] rhttpproxy[0x0053D4A7]
backtrace[07] rhttpproxy[0x0053EE57]
backtrace[08] libc.so.6[0x00022B87]
backtrace[09] rhttpproxy[0x0053B029]
[backtrace end]





Environment

VMware vCenter Server 7.0.0
VMware vCenter Server 8.0

Cause

This happens when the rhttpproxy configuration file /etc/vmware-rhttpproxy/config.xml has been changed incorrectly which prevents the services from starting.

Resolution

Note :  Take a full backup of the vCenter appliance or offline snapshot of the vCenter server

  • To recover from  incorrect changes which prevented the rhttpproxy service from starting should be rolled back
  • If there is no backup or snapshot to revert and if there is no backup copy of the original file. It is possible to validate the xml formatting of the file /etc/vmware-rhttpproxy/config.xml using xmllint validation tool or any preferred method.

     
    For example: 

xmllint --noout --valid /etc/vmware-rhttpproxy/config.xml
parser error : Opening and ending tag mismatch: packetCapture

  • The above output shows that the tag <packetCapture> has been opened but not closed using syntax </packetCapture>


Powered by Wolken Software