vCenter upgrade fails with an error, "Failed to start updatemgr services. Service crashed while starting"

Products

VMware Cloud Foundation

Issue/Introduction

vCenter server upgrade to version 9.1 fails to complete as updatemgr service crashes while attempting to change permission to updatemgr directory and its files. An upgrade performed via SDDC manager or through VAMI, results with similar failure.
This articles provides log snippets that reports about permission attempts, updatemgr service failures and its corresponding backtrace entries, stating the cause and lays out resolutions steps to remediate the situation.
SDDC UI > Upgrade Activity shows vCenter upgrade to failure at "VCENTER UPGRADE INSTALL" stage.

VAMI UI shows installation failure along with error message that reads, "Failed to start updatemgr services. Service crashed while starting"

I

SDDC Manager lcm reports an error along with an error message: "Failed to start updatemgr services. Service crashed while starting" due to "Changing group permission failed with error 256"

/var/log/vmware/vcf/lcm/lcm-debug.log

YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_lcm,####,ea05] [c.v.evo.sddc.lcm.model.task.SubTask,http-nio-127.0.0.1-7400-exec-1] Upgrade error occured:
VCENTER <vCenter FQDN> software packages failed to install. Update info Info (com.vmware.appliance.update.info) => {
state = INSTALL_FAILED,
task = TaskInfo (com.vmware.appliance.task_info) => {
progress = Progress (com.vmware.cis.task.progress) => {
total = 100,
completed = 80,
message = LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = com.vmware.appliance.plain_message,
defaultMessage = Failed to start updatemgr services. Error: Service crashed while starting,
args = [Failed to start updatemgr services. Error: Service crashed while starting],
params = <null>,
localized = <null>

Taking it further in understanding more about updatemgr service crash, vCenter server logs reports the cause is due to an attempt made to change permission of hostupdate directory

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

YYYY-MM-DDTHH:MIN:SEC ERROR applmgmt-update-microservice 1497492 [vc@4413 module="update_b2b_target" funcName="postInstallHook" lineNum="2621"] Failed to start updatemgr services. Error: Service crashed while starting
YYYY-MM-DDTHH:MIN:SEC DEBUG applmgmt-update-microservice 1497492 [vc@4413 module="task_manager_target" funcName="update" lineNum="103"] UpdateTask: status=FAILED, progress=80, message={'id' : 'com. vmware. appliance.plain_message', 'defa
ult message': 'Failed to start updatemgr services. Error: Service crashed while starting', 'args': ['Failed to start updatemgr services. Error: Service crashed while starting']), failure_state=RESUMABLE_AND_REVERTIBLE auto_rollback_state None
YYYY-MM-DDTHH:MIN:SEC INFO vmware-vum-server #### [vc@4413 sub="VciUtilPosix"] [vciUtilPosix 405] SetGroupPermissionOnHostUpdateFolder invoked
YYYY-MM-DDTHH:MIN:SEC INFO vmware-vum-server ####[vc@4413 sub="VciUtilPosix"] [vciUtilPosix 422] /storage/updatemgr/patch-store/hostupdate folder exists, setting group to it
YYYY-MM-DDTHH:MIN:SEC INFO vmware-vum-server ####[vc@4413 sub="VciUtilPosix"] [vciUtilPosix 377] ExecuteGroupPermissionCommand: Executing system command: chgrp -R vum-filedownload-group /storage/updatemgr/patch-store/hostupdate
YYYY-MM-DDTHH:MIN:SEC ERROR vmware-vum-server ####[vc@4413 sub="VciUtilPosix"] [vciUtilPosix 381] Error in executing system command. Return value: 256
YYYY-MM-DDTHH:MIN:SEC ERROR vmware-vum-server ####[vc@4413 sub="VcIntegrity"] [vcIntegrity 775] Error is start vum server :Changing group permission failed with error 256
YYYY-MM-DDTHH:MIN:SEC ERROR vmware-vum-server ####[vc@4413 sub="VcIntegrityPlugin"] [plugin 280] [backtrace begin] product: VMware Update Manager, version: 9.1.0, build: build-25370922, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0047DB64]
--> backtrace[01] libvmacore.so[0x0036D1AC]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x0037E619]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvci-vcIntegrity.so[0x006F19B4]: Integrity::VcIntegrityPlugin::Start()
--> backtrace[04] libvmacore.so[0x00404467]
--> backtrace[05] updatemgr[0x0000E9CF]
--> backtrace[06] updatemgr[0x0000F0AD]
--> backtrace[07] updatemgr[0x0000D66A]
--> backtrace[08] updatemgr[0x0001019C]
--> backtrace[09] libc.so.6[0x0002724A]
--> backtrace[10] libc.so.6[0x00027305]
--> backtrace[11] updatemgr[0x0000D029]
--> backtrace[12] (no module)
--> [backtrace end]
YYYY-MM-DDTHH:MIN:SEC ERROR vmware-vum-server 1560198 [vc@4413 sub="Default"] [main,400] VcIntegrityPlugin::Start unable to start updatemgr. All retries failed.

Environment

VCF 9.1

Cause

Updatemgr requires all files within its ownership. Any file/s that holds permission or privileges besides updatemgr, the service shall fail to start. Usually, the file shall hold higher privileges than the updatemgr. Hence, the command 'chgrp' shall fail as reported in the log snippets and updatemgr services shall crash with a backtrace.

Resolution

Note: Backup of vCenter server is required. Please take valid backup before proceeding further. For more information, refer to Broadcom article Snapshot Best practices for vCenter Server Virtual Machines

Login to vCenter server as 'root' user
Download attached script file "FixPermission.py" from this article and upload the same to vCenter server using winscp or similar application to a /tmp location.
Navigate to /tmp location and give permission using the command chown 777 <FixPermission.py>
Execute the script to identify the ownership of the files
1. python3 FixPermission.py /storage/updatemgr/patch-store > /tmp/file_permission.txt
2. The script shall check for permissions of all files under /storage/updatemgr/patch-store/hostupdate and it shall also apply the change in permission by executing the command chown -R updatemgr:vum-filedownload-group
Read the output file "/tmp/file_permission.txt"
1. Status shows as success, similar to below example
  Running: chown -R updatemgr:vum-filedownload-group /storage/updatemgr/patch-store/hostupdate
  Status: success
2. If the script fails, the output will be similar to
  Running: chown -R updatemgr:vum-filedownload-group /storage/updatemgr/patch-store/hostupdate
  Status: error
  Error number: 1 (EPERM)
  Details: chown: changing ownership of '/storage/updatemgr/patch-store/hostupdate/new.txt': Operation not permitted
Change the privilege of the file manually by running the command:
1. $ sudo chown updatemgr:updatemgr /storage/updatemgr/patch-store/<name of the file>
If the status is successful from either, Step 5a or 6a, proceed to restart vCenter upgrade task from SDDC UI.
If any of the above step fails, consult Broadcom Support for further assistance in reference to this article. (Refer to additional information section of this article for more details)

Additional Information

If the issue persists, raise a Support Request in reference to this article with following details:

Screenshot of the error message.
Log bundle of SDDC Manager Collecting diagnostic information from VMware SDDC Manager
Log bundle of vCenter server Collecting diagnostic information for VMware vCenter Server
Follow the steps mentioned in Broadcom article Creating and managing Broadcom cases

Attachments

fixPermission.py get_app