vpxd-prestart failed with prepareVpxdForNonRootExecution() after rebooting or restarting all services on vCenter

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

vpxd fails to start with the following log:

/var/log/vmware/vmon/vmon*.log

YYYY-MM-DDThh:mm:ss.sssZ Wa(03) host-2862 <vpxd> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862 File "/usr/lib/vmware-vpx/py/vpxd-prestart.py", line 66, in <module>
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862
YYYY-MM-DDThh:mm:ss.sssZ Wa(03) host-2862 <vpxd> Service pre-start command's stderr: prepareVpxdForNonRootExecution()
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862 File "/usr/lib/vmware-vpx/py/utils.py", line 236, in prepareVpxdForNonRootExecution
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862
YYYY-MM-DDThh:mm:ss.sssZ Wa(03) host-2862 <vpxd> Service pre-start command's stderr: setupVpxdServiceAccount()
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862 File "/usr/lib/vmware-vpx/py/utils.py", line 164, in setupVpxdServiceAccount
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862
YYYY-MM-DDThh:mm:ss.sssZ Wa(03) host-2862 <vpxd> Service pre-start command's stderr: new_password = resetServiceAccountPassword(ACCT_NAME)
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862 File "/usr/lib/vmware-vpx/py/utils.py", line 143, in resetServiceAccountPassword
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862
YYYY-MM-DDThh:mm:ss.sssZ Wa(03) host-2862 <vpxd> Service pre-start command's stderr: reset_result = password_stub.reset(Password.ResetSpec(target_username))
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862 File "/usr/lib/vmidentity/svcaccountmgmt/python-bindings/com/vmware/vcenter/svcaccountmgmt_client.py", line 376, in reset
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862
YYYY-MM-DDThh:mm:ss.sssZ Wa(03) host-2862 <vpxd> Service pre-start command's stderr: return self._invoke('reset',
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862 File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862
YYYY-MM-DDThh:mm:ss.sssZ Wa(03) host-2862 <vpxd> Service pre-start command's stderr: File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 295, in native_invoke
YYYY-MM-DDThh:mm:ss.sssZ Wa(03)+ host-2862
YYYY-MM-DDThh:mm:ss.sssZ Wa(03) host-2862 <vpxd> Service pre-start command's stderr: com.vmware.vapi.std.errors_client.Unauthorized: {messages : [LocalizableMessage(id='com.vmware.vapi.authorization.permission.denied', default_message='Insufficient privileges. Contact the Administrator to get the required privileges.', args=[], params=None, localized=None)], data : None, error_type : UNAUTHORIZED}

Environment

vCenter Server 8.x

Cause

/var/cache/svcaccounts/vpxd-svc-acct/.vpxd-svc-acct.new exists unexpectedly.

Normally, .vpxd-svc-acct.new would be created and deleted automatically on the password renewal process of vpxd service account.
However, if the file remains unexpectedly and vpxd is started as part of startup on all services, pre-start process fails to handle the file properly.

Resolution

Note: before any modification to vCenter server, please do backup or snapshot. 

To resolve the issue

1. delete /var/cache/svcaccounts/vpxd-svc-acct/.vpxd-svc-acct.new

# rm /var/cache/svcaccounts/vpxd-svc-acct/.vpxd-svc-acct.new

2. Stop and start all the services

# service-control --stop --all
# service-control --start --all