Error "Failed to force refresh TRUSTED_ROOTS, Error : 183"

Products

• VMware vCenter Server

Issue/Introduction

Upgrading a Platform Service Controller (PSC) or vCenter Server Appliance (vCenter Server) to versions 6.5, 6.7, or 8.x fails during the vmafd firstboot phase.

Symptoms:

• Error message: Failed to force refresh TRUSTED_ROOTS; Error code - 183.

• Upgrade fails during the execution of vecs-cli force-refresh.

• Log files /var/log/firstboot/vmafd-firstboot.py_xxxxx_stderr.log or vmafdd.log contain Error - 1006 or VecsSrvAddCertificate returning 1006.

• This may occur during a converge operation when migrating from external PSC to an embedded deployment.

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 8.x

Cause

The issue is caused by an unexpectedly high number of certificate entries residing in the TRUSTED_ROOT_CRLS store within the VMware Endpoint Certificate Store (VECS).

Resolution

To resolve this issue, the extra entries in the TRUSTED_ROOT_CRLS store must be removed.

1. Log in to the PSC or vCenter Server node via SSH.

2. Verify the number of entries in the TRUSTED_ROOTS store:

   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS | grep Number
   

3. Verify the number of entries in the TRUSTED_ROOT_CRLS store:

   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS | grep Number
   

4. Download the crl-fix.sh script attached and upload it to the /tmp folder of the impacted appliance.

5. Change directory to /tmp:

   cd /tmp
   

6. Make the script executable:

   chmod +x crl-fix.sh
   

7. Remove DOS return characters from the script:

   sed -i -e 's/\r$//' crl-fix.sh
   

8. Execute the script:

   ./crl-fix.sh
   

9. Restart services on all vCenter Servers and PSCs within the same SSO domain.

Additional Information

For instructions on removing expired CA certificates specifically from the TRUSTED_ROOTS store, refer to Broadcom KB 326288.