Error "Failed to force refresh TRUSTED_ROOTS, Error : 183"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Upgrading Platform Service Controller (PSC) to 6.5/6.7 fails with vmafd firstboot with error: Failed to force refresh TRUSTED_ROOTS; Error code - 183 
• Embedded PSC or external PSC upgrade fails with vmafd firstboot error code 183
• /var/log/firstboot/vmafd-firstboot.py_xxxxx_stderr.log

VMAFD boot failed directly after attempt to vecs-cli ‘force refresh’
<timestamp> Running command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'force-refresh']
<timestamp>
<timestamp> VMAFD Boot failed
Exception: Traceback (most recent call last):
 File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 131, in main
   controller.client_mode()
 File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 79, in client_mode
   service.client_mode()
 File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmafdUpgrade.py", line 273, in client_mode
   self.post_init()
 File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmafdUpgrade.py", line 257, in post_init
   self.vecs_force_refresh()
 File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmafdInstall.py", line 748, in vecs_force_refresh
   problemId = "install.vmafd.vecs_force_refresh_failed")
BaseInstallException: {
 "resolution": {
     "id": "install.vmafd.vecs_force_refresh_failed.resolution",
     "localized": "Please search of these symptoms in the VMware Knowledge Base for any known issues and possible workarounds. If none can be found, please collect a support bundle and open a support request.",
     "translatable": "Please search of these symptoms in the VMware Knowledge Base for any known issues and possible workarounds. If none can be found, please collect a support bundle and open a support request."
 },
 "detail": [
     {
         "args": [
             183
         ],
         "id": "install.vmafd.vecs_force_refresh_failed",
         "localized": "Failed to force refresh TRUSTED_ROOTS, Error : 183",
         "translatable": "Failed to force refresh TRUSTED_ROOTS, Error : %(0)d"
     }
 ],
 "componentKey": "vmafd",
 "problemId": "install.vmafd.vecs_force_refresh_failed"
}

• The above symptoms may occur during a converge operation when attempting to migrate from an external PSC to embedded deployment
• vmafdd.log
  <timestamp> err vmafdd t@140029373015808: [Error - 1006, ../../common/ssl.c:1964]
  <timestamp> err vmafdd t@140029373015808: [Error - 1006, ../../../server/vmafd/vecsserviceapi.c:717]
  <timestamp> err vmafdd t@140029373015808: VecsSrvAddCertificate returning 1006
  <timestamp> err vmafdd t@140029373015808: [Error - 1006, ../../../server/vmafd/rootfetch.c:270]
  <timestamp> notice vmafdd t@140029373015808: Failed to update trusted roots. Error [1006]

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 8.x

Cause

This is caused due to the unexpectedly high number of cert entries in TRUSTED_ROOT_CRLS store

Resolution

To workaround the issue, remove the extra entries in the TRUSTED_ROOT_CRLS store

• Login to the PSC node via ssh
• Capture the number of entries in the TRUSTED_ROOTS store

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS | grep Number

• Capture the number of entries in the TRUSTED_ROOT_CRLS store

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS | grep Number

• Download the "crl-fix.sh" script attached to this kb and upload to the impacted PSC or VC with embedded PSC in the /tmp folder, or copy its contents to a text file on the appliance using vi
• cd to /tmp folder
• Run chmod +x crl-fix.sh to make the file executable
• Run sed -i -e 's/\r$//' crl-fix.sh to remove dos return carridges
• Run ./crl-fix.sh
• Restart services on all vCenters and/or PSCs in your SSO domain

Note: To remove expired certificates in TRUSTED_ROOTS, refer to Removing Expired CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store (VECS)