vmware-rbd-watchdog service fails to start with "Could not retrieve certificate and key for service: vpxd-extension"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• When we start the vmware-rbd-watchdog service  it will imidately fails  we see following error in the logs 
• In rbd-syslog.log, You see entries similar to:

BadStatusLine: ''
19-06-26T12:53:40.235327+00:00 err rbd  [49954]logutil:ERROR:IOLogger :
19-06-26T12:53:42.960441+00:00 warning rbd  [4505]rbd_watchdog_linux:WARNING:daemon rbd-cgi with pid 4794 did not respond to signal 2
19-06-26T12:57:14.126833+00:00 err rbd  [24597]vmcacertutil:ERROR:Could not retrieve certificate and key for service: vpxd-extension
rc: 0
out: Error Code : 5

err: Operation Failed: unidentifiable C++ exception
19-06-26T12:57:14.264208+00:00 err rbd  [24597]rbd_watchdog_linux:ERROR:Unable to get client side certificate and key
Traceback (most recent call last):
  File "/build/mts/release/bora-11347054/bora/build/vcenter/release/pylib/rbd_watchdog_linux.py", line 256, in main
  File "/build/mts/release/bora-11347054/bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/vmcacertutil.py", line 166, in getClientCertAndKey
Exception: 0:Error Code : 5
:Operation Failed: unidentifiable C++ exception
19-06-26T12:59:21.028810+00:00 err rbd  [38171]vmcacertutil:ERROR:Could not retrieve certificate and key for service: vpxd-extension
rc: 0
out: Error Code : 5

• In vmafdd-syslog.log, you see entries similar to:

19-06-27T14:21:00.577034+00:00 err vmafdd  t@139892554188544: [Error - 5, ../../../server/vmafd/authutil.c:1055]
19-06-27T14:21:00.577449+00:00 err vmafdd  t@139892554188544: [Error - 5, ../../../server/vmafd/authutil.c:356]
19-06-27T14:21:00.577655+00:00 err vmafdd  t@139892554188544: [Error - 5, ../../../server/vmafd/ipclocalapi.c:868]
19-06-27T14:21:00.577851+00:00 err vmafdd  t@139892554188544: ERROR! VecsIpcGetEntryByAlias failed. Exiting with error : [5]
19-06-27T14:21:00.723995+00:00 err vmafdd  t@139892554188544: [Error - 5, ../../../server/vmafd/authutil.c:1055]
19-06-27T14:21:00.724373+00:00 err vmafdd  t@139892554188544: [Error - 5, ../../../server/vmafd/authutil.c:356]
19-06-27T14:21:00.724613+00:00 err vmafdd  t@139892554188544: [Error - 5, ../../../server/vmafd/ipclocalapi.c:868]
19-06-27T14:21:00.724812+00:00 err vmafdd  t@139892554188544: ERROR! VecsIpcGetEntryByAlias failed. Exiting with error : [5]
 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.5.x

Resolution

To solve this issue add read permission for "deploy" user on vpxd-extension store:

1. To add the permission  : 

/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user deploy --grant read

2. To check permission run the below command 

/usr/lib/vmware-vmafd/bin/vecs-cli  store get-permissions --name vpxd-extension --server localhost --upn [email protected]
 

Example output : 
 

PERMISSIONS FOR STORE: [vpxd-extension]
OWNER : root
USER            ACCESS
deploy             read
updatemgr       read
vsphere-ui        read
vpxd                 read
vsphere-client  read
vsm                  read
imagebuilder    read
content-library read
eam                 read
mbcs               read