VCSA patch upgrade from 8.0.2 to 8.0.3 failed due to an error while creating svcaccount vpxd-svcs-user

Products

VMware vCenter Server

Issue/Introduction

Attempts to upgrade a vCenter 8.0.2 appliance to 8.0.3 fail when all services do not start after reboot of the vCenter appliance during the upgrade process.

The /var/log/vmware/applmgmt/patchrunner.log contains errors starting the vpxd service or other services:

YYYY-MM-DDTHH:MM:SS INFO service_manager Executing command '[['/bin/service-control', '--start', 'vmware-vpxd-svcs']]'
YYYY-MM-DDTHH:MM:SS INFO service_manager Command '[['/bin/service-control', '--start', 'vmware-vpxd-svcs']]' has exit-code='1' and stdout: Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd-svcs...

stderr: Error executing start on service vpxd-svcs. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }

Within /var/log/vmware/vmon/vmon.log, we see error 252 & error 40700: Unknown (private extension) while creating the service account:

YYYY-MM-DDTHH:MM:SS In(05) host-1x2x3x Received start request for vpxd-svcs
YYYY-MM-DDTHH:MM:SS In(05) host-1x2x3x <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
YYYY-MM-DDTHH:MM:SS Wa(03) host-1x2x3x <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 205, in <module>
YYYY-MM-DDTHH:MM:SS Wa(03) host-1x2x3x <vpxd-svcs> Service pre-start command's stderr:     setup_service_account(VPXD_SVCS_SERVICE_ACCOUNT)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 589, in setup_service_account
YYYY-MM-DDTHH:MM:SS Wa(03) host-1x2x3x <vpxd-svcs> Service pre-start command's stderr:     svc_user, svc_password = svcaccount.create_svcaccount(name)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 150, in create_svcaccount
YYYY-MM-DDTHH:MM:SS Wa(03) host-1x2x3x <vpxd-svcs> Service pre-start command's stderr:     raise InvokeCommandException(error)
YYYY-MM-DDTHH:MM:SS Wa(03) host-1x2x3x <vpxd-svcs> Service pre-start command's stderr: cis.exceptions.InvokeCommandException: {
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x     "detail": [
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x         {
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x             "id": "install.ciscommon.command.errinvoke",
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x             "translatable": "An error occurred while invoking external command : '%(0)s'",
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x             "args": [
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x                 "Error 252 while creating svcaccount \"vpxd-svcs-user-########-####-####-####-############\":\ndir-cli failed. Error 40700: Unknown (private extension) error \n"
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x             ],
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x            "localized": "An error occurred while invoking external command : 'Error 252 while creating svcaccount \"vpxd-svcs-user-########-####-####-####-############\":\ndir-cli failed. Error 40700: Unknown (private extension) error \n'"
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x        }
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x     ],
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x     "componentKey": null,
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x     "problemId": null,
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x     "resolution": null
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-1x2x3x }
YYYY-MM-DDTHH:MM:SS Er(02) host-1x2x3x <vpxd-svcs> Service pre-start command failed with exit code 1.
YYYY-MM-DDTHH:MM:SS Wa(03) host-1x2x3x [ReadSvcSubStartupData] No startup information from vpxd-svcs.

Within /var/log/vmware/vpxd-svcs/pre-start-vpxd-svcs.log, we see the same issue with the creation of the service account and failure of the vpxd service to start:

YYYY-MM-DDTHH:MM:SS INFO pre-start-vpxd-svcs Hash computed for the service account not same as previously stored hash.
YYYY-MM-DDTHH:MM:SS INFO pre-start-vpxd-svcs Setting-up service-account for service: vpxd-svcs-user
YYYY-MM-DDTHH:MM:SS INFO pre-start-vpxd-svcs Remove hash file: /var/cache/svcaccounts/vpxd-svcs-user/vpxd-svcs-user.hash
YYYY-MM-DDTHH:MM:SS INFO pre-start-vpxd-svcs /var/cache/svcaccounts/vpxd-svcs-user/vpxd-svcs-user.hash does not exist, skipping delete.
YYYY-MM-DDTHH:MM:SS INFO pre-start-vpxd-svcs Service account for vpxd-svcs-user is invalid, Will go ahead with service account recreation.
YYYY-MM-DDTHH:MM:SS INFO pre-start-vpxd-svcs Service-account vpxd-svcs-user-########-####-####-####-############ does not exist

Within /var/log/vmware/likewise/likewise.log, we see the likewise service is starting and crashing during this process:

YYYY-MM-DDTHH:MM:SS INFO netlogon: [0x7fd06c944740] LWNET Service exiting...
YYYY-MM-DDTHH:MM:SS INFO lwsm: Stopping service: lwreg
YYYY-MM-DDTHH:MM:SS WARNING lwreg: [0x7f8bec157740] Received signal [15]
YYYY-MM-DDTHH:MM:SS INFO lwreg-ipc: [0x7f8bec157740] Listener shut down
....
YYYY-MM-DDTHH:MM:SS WARNING lwsm: Restarting dead service: lwreg (attempt 1/2)
YYYY-MM-DDTHH:MM:SS INFO lwsm: Starting service: lwreg

Environment

vCenter patch update from 8.0.2 to 8.0.3

Cause

Errors within the likewise/LDAP configuration for the vCenter prevent the vCenter from properly allocation service IDs and starting services on the vCenter appliance.

Resolution

Before proceeding with the steps below, take both a backup and a snapshot of the vCenter Server Appliance. If the vCenter is part of a Enhanced Linked Mode (ELM) replication setup, also take a backup or offline (powered off) snapshot of all replicating vCenter ELM nodes.

Warning: Active directory users and permissions may need to be manually re-added to the vCenter appliance after the upgrade. Please record any custom assignments, rules, roles, or permissions related to Active Directory users and groups in case they need to be manually re-applied to the vCenter AD and Identity Providers configuration.

Workaround:

Remove the vCenter from AD and any external identity sources
See the following documentation for instructions on removing a vCenter appliance from and active directory domain: Join or Leave an Active Directory Domain
See the following documentation for instructions on removing an Identity Provider from a vCenter appliance: Remove a vCenter Single Sign-On Identity Source
Re-attempt the upgrade
Once the update is successfully applied, re-add the vCenter to the Active Directory domain and re-add the external Identity Sources.

NOTE: Should the upgrade fail after the AD configuration and Identity Sources were removed, with the same or similar errors, please export a support bundle from the vCenter appliance while it is still in a failed state (Collecting diagnostic information for VMware vCenter Server 7.x and 8.x) and open a support request with Broadcom or your Partner Services Provider, then roll back the upgrade to your previous snapshot or backup and upload the support bundle to your SR.