vCenter upgrade failure from version 7 to 8 during certificate management firstboot

Products

• VMware vCenter Server

Issue/Introduction

vCenter upgrade fails while certificate management firstboot while upgrading from version 7 to 8 with error  as below:

Error An error occurred while starting service 'certificatemanagement'
Resolution This is an unrecoverable error, please retry install. If you encounter this error again, please search for
 these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If
 none can be found, collect a support bundle and open a support request.

 

Log reference:

/var/log/vmware/vmon/vmon.log

"Error 159 while adding user cms-XXXXXXXXXXXXXX to SSO group \"CAAdmins\":\ndir-cli failed. Error 5023: Possible errors: \nLDAP error: Unknown (extension) error \nWin Error: Operation failed with error ERROR_INVALID_STATE (5023) \n"

/var/log/firstboot/firstbootInfrastructure.log
INFO firstbootInfrastructure [Failed] /usr/lib/vmware-certificatemanagement/firstboot/certificatemanagement_firstboot.py is complete
WARNING firstbootInfrastructure Bug component info file does not exist
 INFO firstbootInfrastructure Firstboot duration: 398 sec
INFO firstbootInfrastructure First boot is a failure
ERROR firstbootInfrastructure Installation of vCenter server failed with firstboot scripts

/var/log/firstboot/certificatemanagement_firstboot.py_14802_stdout.log
Replacing properties in /usr/lib/vmware-certificatemanagement/config/certificatemanagement.properties
Starting CertificateManagement Service
Starting certificatemanagement

/var/log/firstboot/certificatemanagement_firstboot.py_14802_stderr.log
 INFO 67 Certificate Management FBActions.FIRSTBOOT
INFO 156 Version found: (before, 0)
DEBUG 2491 vc desired state is '('default', '{"services":{"absent":[]}}')'
 ERROR starting certificatemanagement rc: 4, stdout: , stderr: Start service request failed. Error: A system error occurred. Check logs for details

ERROR 212 Certificate Management Firstboot failed
ERROR 213 Exception: Traceback (most recent call last):
  File "/usr/lib/vmware-certificatemanagement/firstboot/certificatemanagement_firstboot.py", line 200, in main
    certMgrFb.firstbootAction()
  File "/usr/lib/vmware-certificatemanagement/firstboot/certificatemanagement_firstboot.py", line 75, in firstbootAction
    self.startService()
  File "/usr/lib/vmware-certificatemanagement/firstboot/certificatemanagement_firstboot.py", line 51, in startService
    self.start_service()
  File "/usr/lib/vmware/site-packages/cis/firstboot.py", line 241, in start_service
    service_start(self.get_eff_service_name())
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 1173, in service_start
    raise ServiceStartException(svc_name)
cis.exceptions.ServiceStartException: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "certificatemanagement"
            ],
            "localized": "An error occurred while starting service 'certificatemanagement'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

Traceback (most recent call last):
  File "/usr/lib/vmware-certificatemanagement/firstboot/certificatemanagement_firstboot.py", line 200, in main
    certMgrFb.firstbootAction()
  File "/usr/lib/vmware-certificatemanagement/firstboot/certificatemanagement_firstboot.py", line 75, in firstbootAction
    self.startService()
  File "/usr/lib/vmware-certificatemanagement/firstboot/certificatemanagement_firstboot.py", line 51, in startService
    self.start_service()
  File "/usr/lib/vmware/site-packages/cis/firstboot.py", line 241, in start_service
    service_start(self.get_eff_service_name())
File "/usr/lib/vmware/site-packages/cis/utils.py", line 1173, in service_start
    raise ServiceStartException(svc_name)
cis.exceptions.ServiceStartException: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "certificatemanagement"
            ],
            "localized": "An error occurred while starting service 'certificatemanagement'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

/var/log/vmware/certificatemanagement/certificatemanagement_prestart.log
 INFO certificatemanagement_prestart Service-account password file does not exist
ERROR certificatemanagement_prestart Not found service account hash file: /var/cache/svcaccounts/cms/cms.hash
INFO certificatemanagement_prestart Hash computed for the service account not same as previously stored hash.
 INFO certificatemanagement_prestart Setting-up service-account for service: cms
 INFO certificatemanagement_prestart Remove hash file: /var/cache/svcaccounts/cms/cms.hash
INFO certificatemanagement_prestart /var/cache/svcaccounts/cms/cms.hash does not exist, skipping delete.
INFO certificatemanagement_prestart Service account for cms is invalid, Will go ahead with service account recreation.
 INFO certificatemanagement_prestart Service-account cms-XXXXXXXXX does not exist
INFO certificatemanagement_prestart Service-account cms-XXXXXXXXX created successfully

Environment

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

 

Cause

Presence of duplicate CAAdmins under SSO group.

Resolution

To resolve the issue, Remove the duplicate entry using the below steps:

1. SSH into the appliance shell of the vCenter.

2. cd /

3. Run the below command :

ldapdelete -H ldap://localhost -x -D "cn=administrator,cn=users,dc=<vcentername>,dc=<domain>" -W "CN=CAAdmins,dc=<vcentername>,dc=<domain>"

4.Restart vmdird (optional), if standalone.

service-control --restart vmdird