VMware vCenter Server upgrade Fails with: Encountered an internal error during firstboot

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter Server upgrade fails during firstboot with error Encountered an internal error
• You see similar log entries as:

/var/log/firstboot/fbInstall.json :

{
    "progress": 22,
    "progress_message": {
        "id": "install.ciscommon.component.starting",
        "translatable": "Starting %(0)s...",
        "args": [
            "VMware Security Token Service"
        ],
        "localized": "Starting VMware Security Token Service..."
    },
    "status": "error",
    "info": [],
    "warning": [],
    "question": null,
    "error": {
        "detail": [
            {
                "id": "install.ciscommon.internal.error",
                "translatable": "Encountered an internal error.\n\n%(0)s",
                "args": [
                    "Traceback (most recent call last):\n  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 249, in securityctx_modifier\n    yield\n  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 348, in add_securityctx_to_requests\n    return req_method(self, *args, **kargs)\n  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 358, in register_service\n    svc_create_spec)\n  File \"/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py\", line 556, in <lambda>\n    self.f(*(self.args + (obj,) + args), **kwargs)\n  File \"/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py\", line 368, in _InvokeMethod\n    return self._stub.InvokeMethod(self, info, args)\n  File \"/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py\", line 1481, in InvokeMethod\n    raise obj # pylint: disable-msg=E0702\npyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) {\n   dynamicType = <unset>,\n   dynamicProperty = (vmodl.DynamicProperty) [],\n   msg = '',\n   faultCause = <unset>,\n   faultMessage = (vmodl.LocalizableMessage) []\n}\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py\", line 1752, in main\n    vmidentityFB.boot()\n  File \"/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py\", line 367, in boot\n    self.registerTokenServiceWithLookupService()\n  File \"/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py\", line 654, in registerTokenServiceWithLookupService\n    raise e\n  File \"/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py\", line 650, in registerTokenServiceWithLookupService\n    dynVars=dynVars)\n  File \"/usr/lib/vmware-cm/bin/cloudvmcisreg.py\", line 710, in cloudvm_sso_cm_register\n    serviceId = do_lsauthz_operation(cisreg_opts_dict)\n  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 1116, in do_lsauthz_operation\n    ls_obj.register_service(svc_id, svc_create_spec)\n  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 348, in add_securityctx_to_requests\n    return req_method(self, *args, **kargs)\n  File \"/usr/lib/python3.7/contextlib.py\", line 161, in __exit__\n    raise RuntimeError(\"generator didn't stop after throw()\")\nRuntimeError: generator didn't stop after throw()\n"
                ],

vmidentity-firstboot.py_51473_stderr.log:
[YYYY-MM-DDTHH:MM:SS] Security error: (vmodl.fault.SecurityError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = '',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) []
}

[YYYY-MM-DDTHH:MM:SS] Failed to register VMware Token Service with Lookup Service.

Environment

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

Cause

This issue can happen if SSO Administrator account is not authorized to add service to the Lookup Service. This may be because SSO Administror is not available in Builtin Administrator group in VMDIRD.

Resolution

To resolve this issue:

Connect to the vSphere Single-Sign on VMDir database referring to the article Using JXplorer to connect to the vSphere Single Sign-on



Verify that the SSO Administrator user is missing under Builtin > Administrators.

To add the user:

1. Go to Builtin - Administrators > Table Editor
2. Right click on member space and click on Add another Value
3. On the value field, add : cn=Administrator,cn=Users,dc=vsphere,dc=local
   
   If SSO domain name is not vsphere.local, change it accordingly.
   
   
   
   Submit the changes and run the upgrade again.