vcsa upgrade fails with with vmware-updatemgr failed to start
search cancel

vcsa upgrade fails with with vmware-updatemgr failed to start

book

Article ID: 315230

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This article provides a solution to upgrade vcenter appliance when facing the previous Symptoms .

Symptoms:
1- vcsa upgrade fails with with vmware-updatemgr failed to start 

2- Import_com.vmware.vcIntegrity log in /var/log/vmware/upgrade shows update manager failed to start directly after successfully finishing the import phase with the below trace :

<timestamp> INFO vcIntegrity_upgrade Upgrade Database Schema StdErr:
<timestamp> INFO vcIntegrity_upgrade
<timestamp> INFO vcIntegrity_upgrade Upgrade Database Schema completed
<timestamp> INFO vcIntegrity_upgrade vcIntegrity Import phase. Importing data destination completed.


and VUM service is failing to start after Import : 

<timestamp>  INFO vcIntegrity_upgrade Start vmware-updatemgr service
<timestamp>  INFO service_manager Executing command '[['/bin/service-control', '--status', 'vmware-updatemgr']]'
<timestamp>  INFO service_manager Command '[['/bin/service-control', '--status', 'vmware-updatemgr']]' has exit-code='0' and stdout: Stopped:
 vmware-updatemgr


stderr:

<timestamp> INFO service_manager Starting service 'vmware-updatemgr' ...
<timestamp>  INFO service_manager Executing command '[['/bin/service-control', '--start', 'vmware-updatemgr']]'
<timestamp>  INFO service_manager Command '[['/bin/service-control', '--start', 'vmware-updatemgr']]' has exit-code='1' and stdout: Operation not cancellable. Please wait for it to finish...
Performing start operation on service updatemgr...


stderr: Error executing start on service updatemgr. Details {
    "resolution": null,
    "componentKey": null,
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "args": [
                "updatemgr"
            ],
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'updatemgr'"
        }
    ],
    "problemId": null
}
Service-control failed. Error: {
    "resolution": null,
    "componentKey": null,
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "args": [
                "updatemgr"
            ],
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'updatemgr'"
        }
    ],
    "problemId": null
}

<timestamp>  ERROR __main__ Upgrade Phase 'vcIntegrity:Import' failed. Exception: Service cannot be started. Error: Error executing start on service updatemgr. Details {
    "resolution": null,
    "componentKey": null,
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "args": [
                "updatemgr"
            ],
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'updatemgr'"
        }
    ],
    "problemId": null
}
Service-control failed. Error: {
    "resolution": null,
    "componentKey": null,
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "args": [
                "updatemgr"
            ],
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'updatemgr'"
        }
    ],
    "problemId": null
}

Traceback (most recent call last):
  File "/usr/lib/vmware/cis_upgrade_runner/payload/componentPhaseLauncher.py", line 461, in main
    executionResult = systemExtension(exeContext)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/extensions.py", line 94, in __call__
    result = self.extension(*args)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/extensions.py", line 110, in _func
    return func(*args)
  File "/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/vcIntegrity/__init__.py", line 60, in doImport
    return vcIntegrity_upgrade.importData(context)
  File "/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/vcIntegrity/vcIntegrity_upgrade.py", line 511, in importData
    importDataLin(context)
  File "/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/vcIntegrity/vcIntegrity_upgrade.py", line 504, in importDataLin
    serviceManager.start(VUM_SERVICE_NAME)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/service_manager.py", line 979, in wrapper
    return getattr(controller, attr)(*args, **kwargs)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/service_manager.py", line 887, in start
    super(VMwareServiceController, self).start(serviceName)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/service_manager.py", line 745, in start
    raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service updatemgr. Details {
    "resolution": null,
    "componentKey": null,
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "args": [
                "updatemgr"
            ],
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'updatemgr'"
        }
    ],
    "problemId": null
}
Service-control failed. Error: {
    "resolution": null,
    "componentKey": null,
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "args": [
                "updatemgr"
            ],
            "translatable": "An error occurred while starting service '%(0)s'",
            "localized": "An error occurred while starting service 'updatemgr'"
        }
    ],
    "problemId": null
}

3- vpxd.log Shows "SQL execution took too long" error on the events table :


<timestamp>  info vpxd[19014] [Originator@6876 sub=vpxLro opID=#####] [VpxLRO] -- BEGIN lro-34048 -- EventManager -- vim.event.EventManager.createCollector -- ########-####-####-####-############(#######-####-####-####-########)
<timestamp>  warning vpxd[19014] [Originator@6876 sub=Default opID=#####] [VdbStatement] SQL execution took too long:  SELECT EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME, USERNAME, CATEGORY, VM_ID,        VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE,        COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID,        NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, STORAGEPOD_ID, STORAGEPOD_NAME, CHANGE_TAG_ID FROM VPXV_EVENT_ALL  WHERE CREATE_TIME>=? AND (EVENT_TYPE IN (?,?,?,?,?,?,?)) AND EVENT_ID<? ORDER BY EVENT_ID DESC LIMIT 10;
<timestamp>  warning vpxd[19014] [Originator@6876 sub=Default opID=#####] [VdbStatement] Execution elapsed time: 713402 ms
<timestamp>  info vpxd[19014] [Originator@6876 sub=vpxLro opID=#####] [VpxLRO] -- FINISH lro-34048

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x

Cause

Issue is due to having a huge VPX_EVENT table causing update manager start operation to timeout

Resolution

Powered by Wolken Software