"Running firstboot stage 1 of 4 failed.", vCenter Server Convergence fails during firstboot phase

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Converging Platform Service Controller (PSC) with vCenter fails with vmafd firstboot with error: "Running firstboot stage 1 of 4 failed."

 "message": "Running firstboot stage 1 of 4 failed.", 
  {
            "key": "firstbootpart1",
            "value": {
                "description": "Run vmafd firstboot",
                "status": "FAILED"
            }
        },

 

• VMAFD firstboot log file /var/log/firstboot/vmafd-firstboot.py_xxxxx_stderr.log, will show entries similar to :

{

   "translatable": "Failed to force refresh TRUSTED_ROOTS, Error : %(0)d",
   "localized": "Failed to force refresh TRUSTED_ROOTS, Error : 238",
   "args": [
        238
   ],
    "id": "install.vmafd.vecs_force_refresh_failed"
}

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x

Cause

This is caused due to the unexpectedly high number of certificate entries in TRUSTED_ROOT_CRLS VECS store.

Resolution

To resolve the issue, remove the extra entries in the TRUSTED_ROOT_CRLS store.

• Login to the PSC node via ssh
• Capture the number of entries in the TRUSTED_ROOT_CRLS store

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS | grep Number

Take snapshots of the PSCs and the vCenter Servers and continue with below steps to remove the extra entries from TRUSTED_ROOT_CRLS store.

1. Download the "crl-fix.sh" script attached to this kb and upload to the impacted PSC and VC in the /tmp folder, or copy its contents to a text file on the appliance using vi editor
2. cd to /tmp folder
3. Run chmod +x crl-fix.sh to make the file executable
4. Run ./crl-fix.sh
5. Restart the services: service-control --stop --all && service-control --start --all
6. Repeat Steps 1-5 on the vCenter Server
7. Verify the number of entries in TRUSTED_ROOT_CRLS and make sure the extra entries are removed, ideally it will be in single digits
   • ​​​​​​​/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS | grep Number
8. Proceed with Convergence