The vCenter rhttpproxy service fails to start and does not log anything to it's log file(s)
Article ID: 381199
Updated On:
Products
Issue/Introduction
This article applies when the rhttpproxy service (or potentially another vCenter service) fails to start with little to no logging. You may see an error similar to this in "dmesg" or "journalctl -b 0":
traps: rhttpproxy[47473] general protection fault ip:55555555 sp:7ffc54f0d7c8 error:0 in rhttpproxy[55555555+8da000]
Environment
To identify the issue, verify the service RPM (in this case, rhttpproxy):
First, get the name and version of the service RPM:
root@vc [ ~ ]# rpm -qa | grep rhttpproxy
VMware-rhttpproxy-8.0.3.00400-14373555.x86_64
Next, verify the integrity of the RPM:
root@vc [ ~ ]# rpm -V VMware-rhttpproxy-8.0.3.00200-14172000.x86_64
Healthy output you receive should be similar to this:
root@vc [ ~ ]# rpm -V VMware-rhttpproxy-8.0.3.00200-14172000.x86_64
S.5....T. c /etc/vmware-rhttpproxy/config.xml
.....UG.. /etc/vmware-rhttpproxy/endpoints.conf.d/rhttpproxy-stats.conf
.......T. c /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd.conf
.....UG.. /etc/vmware-sca/resourcebundles/rhttpproxyResourceBundle.zip
S.5....T. c /etc/vmware/appliance/firewall/vmware-rhttpproxy
SM5....T. c /etc/vmware/install-defaults/rhttpproxy.cert
SM5....T. c /etc/vmware/install-defaults/rhttpproxy.conf.path
.M......P /usr/lib/vmware-rhttpproxy/rhttpproxy <---- Here
As many of the files can differ in different ways after installation (due to configuration during the firstboot phase), ignore any of the entries that exist outside /usr/lib/.
To interpret the results, use the following key:
S file Size differs
M Mode differs (includes permissions and file type)
5 digest (formerly MD5 sum) differs
D Device major/minor number mismatch
L readlink(2) path mismatch
U User ownership differs
G Group ownership differs
T mTime differs
P caPabilities differ
Check for a digest mismatch on the binary file. This would be an example of a corrupt binary:
root@vc [ ~ ]# rpm -V VMware-rhttpproxy-8.0.3.00200-14172000.x86_64
S.5..U.T. c /etc/vmware-rhttpproxy/config.xml
.....UG.. /etc/vmware-rhttpproxy/endpoints.conf.d/rhttpproxy-stats.conf
.......T. c /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd.conf
.....UG.. /etc/vmware-sca/resourcebundles/rhttpproxyResourceBundle.zip
S.5....T. c /etc/vmware/appliance/firewall/vmware-rhttpproxy
SM5....T. c /etc/vmware/install-defaults/rhttpproxy.cert
SM5....T. c /etc/vmware/install-defaults/rhttpproxy.conf.path
.M5.....P /usr/lib/vmware-rhttpproxy/rhttpproxy <---- Here
From the above output, we can see there is an MD5 digest mismatch for the binary file.
Cause
This is typically caused by underlying storage issues that resulted in partial 'zeroing out' of the binary. This renders the binary non-functional.
Resolution
To resolve the issue, obtain a copy of the binary from a vCenter server of the same version that is known to be healthy. Place the binary in the /tmp directory of the vCenter server.
Before copying the healthy binary to the proper location, backup the existing corrupt binary:
root@vc [ ~ ]# mv /usr/lib/vmware-rhttpproxy/rhttpproxy /usr/lib/vmware-rhttpproxy/rhttpproxy.backup
Copy the healthy binary to the original location:
root@vc [ ~ ]# cp /tmp/rhttpproxy /usr/lib/vmware-rhttpproxy/rhttpproxy
Copy the permissions and ownership from the original binary to the new binary:
root@vc [ ~ ]# chown --reference=/usr/lib/vmware-rhttpproxy/rhttpproxy.backup /usr/lib/vmware-rhttpproxy/rhttpproxy
root@vc [ ~ ]# chmod --reference=/usr/lib/vmware-rhttpproxy/rhttpproxy.backup /usr/lib/vmware-rhttpproxy/rhttpproxy
Verify that the file capabilities are applied as well. To do this, first get the file capabilties on the original binary file (if there are no results from this command, perform the command on the vCenter from which you copied the healthy binary):
root@vc [ ~ ]# getcap /usr/lib/vmware-rhttpproxy/rhttpproxy.backup
/usr/lib/vmware-rhttpproxy/rhttpproxy cap_net_bind_service,cap_net_admin,cap_net_raw=ep
Apply the same file capabilities to the healthy binary:
root@vc [ ~ ]# setcap cap_net_bind_service,cap_net_admin,cap_net_raw=ep /usr/lib/vmware-rhttpproxy/rhttpproxy
Restart all services to ensure the service starts normally:
root@vc [ ~ ]# service-control --stop --all && service-control --start --all
Feedback
