"Running firstboot stage 1 of 4 failed.", vCenter Server Convergence fails during firstboot phase
search cancel

"Running firstboot stage 1 of 4 failed.", vCenter Server Convergence fails during firstboot phase

book

Article ID: 322230

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • Converging Platform Service Controller (PSC) with vCenter fails with vmafd firstboot with error: "Running firstboot stage 1 of 4 failed."
 "message": "Running firstboot stage 1 of 4 failed.", 
  {
            "key": "firstbootpart1",
            "value": {
                "description": "Run vmafd firstboot",
                "status": "FAILED"
            }
        },

 
  • VMAFD firstboot log file /var/log/firstboot/vmafd-firstboot.py_xxxxx_stderr.log, will show entries similar to :
{
   "translatable": "Failed to force refresh TRUSTED_ROOTS, Error : %(0)d",
   "localized": "Failed to force refresh TRUSTED_ROOTS, Error : 238",
   "args": [
        238
   ],
    "id": "install.vmafd.vecs_force_refresh_failed"
}


Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x

Cause

This is caused due to the unexpectedly high number of certificate entries in TRUSTED_ROOT_CRLS VECS store.

Resolution

To resolve the issue, remove the extra entries in the TRUSTED_ROOT_CRLS store.
  • Login to the PSC node via ssh
  • Capture the number of entries in the TRUSTED_ROOT_CRLS store
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS | grep Number


Take snapshots of the PSCs and the vCenter Servers and continue with below steps to remove the extra entries from TRUSTED_ROOT_CRLS store.

  1. Download the "crl-fix.sh" script attached to this kb and upload to the impacted PSC and VC in the /tmp folder, or copy its contents to a text file on the appliance using vi editor
  2. cd to /tmp folder
  3. Run chmod +x crl-fix.sh to make the file executable
  4. Run ./crl-fix.sh
  5. Restart the services: service-control --stop --all && service-control --start --all
  6. Repeat Steps 1-5 on the vCenter Server
  7. Verify the number of entries in TRUSTED_ROOT_CRLS and make sure the extra entries are removed, ideally it will be in single digits
    • ​​​​​​​/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS | grep Number
  8. Proceed with Convergence


Attachments

crl-fix get_app