Upgrade of vCenter from 7.x to 8.x fails around 25% during deployment with error - "A problem occurred while Starting VMware Security Token Service."
search cancel

Upgrade of vCenter from 7.x to 8.x fails around 25% during deployment with error - "A problem occurred while Starting VMware Security Token Service."

book

Article ID: 422806

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vCenter Upgrade to 7.0 fails with an error: A problem occurred while Starting VMware Security Token Service.

A problem occurred while Starting VMware Security Token Service.Encountered an internal error. Traceback (most recent call last): File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1170, in main vmidentityFB.boot() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 275, in boot self.configureSTS(self.__stsRetryCount, self.__stsRetryInterval) File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 791, in configureSTS self.startSTSService() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 751, in startSTSService returnCode = self.startService(self.__sts_service_name) File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 80, in startService update_services_runstate("start", None, False, False, svc_names=[svc_name]) File "/usr/lib/vmware/site-packages/cis/svcsController.py", line 1122, in update_services_runstate _update_services_runstate_svclist('start', svc_nodenames, File "/usr/lib/vmware/site-packages/cis/svcsController.py", line 883, in _update_services_runstate_svclist controller.start_svc(svc_id, explicit_op=explicit_op) File "/usr/lib/vmware/site-packages/cis/svcsController.py", line 516, in start_svc service_start(svc_id, quiet=_quiet, File "/usr/lib/vmware/site-packages/cis/utils.py", line 1173, in service_start raise ServiceStartException(svc_name) cis.exceptions.ServiceStartException: { "detail": [ { "id": "install.ciscommon.service.failstart", "translatable": "An error occurred while starting service 'Traceback (most recent call last): File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1170, in main vmidentityFB.boot() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 275, in boot self.configureSTS(self.__stsRetryCount, self.__stsRetryInterval) File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 791, in configureSTS self.startSTSService() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 751, in startSTSService returnCode = self.startService(self.__sts_service_name) File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 80, in startService update_services_runstate("start", None, False, False, svc_names=[svc_name]) File "/usr/lib/vmware/site-packages/cis/svcsController.py", line 1122, in update_services_runstate _update_services_runstate_svclist('start', svc_nodenames, File "/usr/lib/vmware/site-packages/cis/svcsController.py", line 883, in _update_services_runstate_svclist controller.start_svc(svc_id, explicit_op=explicit_op) File "/usr/lib/vmware/site-packages/cis/svcsController.py", line 516, in start_svc service_start(svc_id, quiet=_quiet, File "/usr/lib/vmware/site-packages/cis/utils.py", line 1173, in service_start raise ServiceStartException(svc_name) cis.exceptions.ServiceStartException: { "detail": [ { "id": "install.ciscommon.service.failstart", "translatable": "An error occurred while starting service '%(0)s'", "args": [ "sts" ], "localized": "An error occurred while starting service 'sts'" } ], "componentKey": null, "problemId": null, "resolution": null } '", "args": [ "sts" ], "localized": "An error occurred while starting service 'sts'" } ], "componentKey": null, "problemId": null, "resolution": null }

This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.

  • When reviewing the content in the STS_INTERNAL_SSL_CERT store in the VECS you find that it contains no entries:

# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store STS_INTERNAL_SSL_CERT
Number of entries in store : 0

Environment

vCenter Server 7.x

vCenter Server 8.x

Cause

When vCenter Server was upgraded from an older version in the past, the installer expects to find an  STS_INTERNAL_SSL_CERTIFICATE store in the VECS, containing the machine SSL certificate.

If this store does not exist, or holds no entries, the vCenter installer will fail to verify the current STS certificate during the pre-check phase and will get stuck.

As a result, when its default timeout is reached, the upgrade fails with the error shown above.

Resolution

To solve this, publish the current machine SSL certificate to the STS_INTERNAL_SSL_CERT store using the following steps:

  1. Ensure that you have a current backup of the vCenter
  2. Open an SSH connection to the old vCenter Server Appliance.
  3. Create an export folder:
    # mkdir /certs
  4. Export the current machine SSL certificate and public key from the MACHINE_SSL_CERT store:

# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --alias __MACHINE_CERT --store MACHINE_SSL_CERT >/certs/__MACHINE_CERT.crt
# /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --alias __MACHINE_CERT --store MACHINE_SSL_CERT >/certs/__MACHINE_CERT.key

     5. Publish the certificate as a new entry called __MACHINE_CERT in the STS_INTERNAL_CERT store:

         # /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert /certs/__MACHINE_CERT.crt --key /certs/__MACHINE_CERT.key

    6.Initiate the upgrade process again.

Additional Information

We can also remove this store using Script for safe removal. Refer to below KB for steps for supported method,

Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH

 

Powered by Wolken Software