Database Initialization Error after vCenter Server upgrade from 6.7 to 7.0

Products

• VMware vCenter Server

Issue/Introduction

• vCenter was migrated from 6.7 to 7.0, and following the migration vmware-updatemgr fails to initialize.
• The Update Manager service crashes with backtraces shortly after starting:

error vmware-vum-server[17668] [Originator@6876 sub=JobDispatcher] [JobDispatcher 104] [backtrace begin] product: VMware Update Manager, version: 7.0.3, build: build-20764999, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0037DA77]
--> backtrace[01] libvmacore.so[0x002C78D3]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x002D6B69]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvci-vcIntegrity.so[0x00406143]: Integrity::JobDispatcher::GetInstance(Integrity::VcIntegrity*, Integrity::BaselineMgrInternal*)
--> backtrace[04] libvci-vapi.so[0x009CD88C]: VumVapi::Utils::InitOnlineAndUmdsDepots()
--> backtrace[05] libvci-vapi.so[0x00624459]: Integrity::VapiPlugin::InitiateDefaultTasks()
--> backtrace[06] libvmacore.so[0x002349A7]
--> backtrace[07] libvmacore.so[0x00239F4F]
--> backtrace[08] libvmacore.so[0x003764AC]
--> backtrace[09] libpthread.so.0[0x00007F87]
--> backtrace[10] libc.so.6[0x000F362F]
--> backtrace[11] (no module)
--> [backtrace end]

Environment

vCenter 7.0

vCenter 8.x

Cause

The Update Manager database import can timeout, causing database corruption which leads to the service failing. Prior to the crash, errors in the `vmware-vum-server.log` point at database problems:

YYYY-MM-DDThh:mm:ss.000Z-05:00 error vmware-vum-server[17510] [Originator@6876 sub=VcIntegrity] [vcIntegrity 833] Error initializing database VdbError: PM_HCL_RESULT_ID_SEQ
YYYY-MM-DDThh:mm:ss.000Z-05:00 error vmware-vum-server[17510] [Originator@6876 sub=VcIntegrity] [vcIntegrity 760] Error is start vum server :Fault cause: vim.fault.DatabaseError
YYYY-MM-DDThh:mm:ss.000Z-05:00 error vmware-vum-server[17510] [Originator@6876 sub=JobDispatcher] [JobDispatcher 104] Bad parameters in JobDispatcher::GetInstance.
YYYY-MM-DDThh:mm:ss.000Z-05:00 error vmware-vum-server[17668] [Originator@6876 sub=JobDispatcher] [JobDispatcher 104] Bad parameters in JobDispatcher::GetInstance.
YYYY-MM-DDThh:mm:ss.000Z-05:00 error vmware-vum-server[17510] [Originator@6876 sub=VumVapiEndpoint] [plugin 744] Cleaning DRS MMode requests failed due to error: Dynamic exception type: Integrity::JobDispatcher::JobDispatcherException
--> std::exception::what: Bad parameters in JobDispatcher::GetInstance.

Upgrade logging in the `/var/log/vmware/upgrade/import.json` indicates that the VUM database import timed out:

    {
    "progress": 32,
    "progress_message": {
    "id": "ur.upgrade.script.import.progress.text",
    "translatable": "Importing %(0)s data...",
    "args": [
    "VMware vSphere Update Manager"
    ],
    "localized": "Importing VMware vSphere Update Manager data..."
    },
    "status": "error",
    "info": [],
    "warning": [],
    "question": null,
    "error": {
    "detail": [
    {
    "id": "ur.upgrade.execution.timedout.text",
    "translatable": "Upgrade phase timed out. The time planned for the upgrade phase was %(0)s minutes. The upgrade phase has already been running for %(1)s minutes.",
    "args": [
    60,
    60
    ],
    "localized": "Upgrade phase timed out. The time planned for the upgrade phase was 60 minutes. The upgrade phase has already been running for 60 minutes."
    }
    ],
    "componentKey": "upgrade_framework",
    "problemId": null,
    "resolution": {
    "id": "ur.upgrade.execution.timedout.resolution",
    "translatable": "To extend the default timeout, set environment variable %(0)s for export upgrade phase, %(1)s for import upgrade phase, and  %(2)s for import historical data phase. The timeout is in minutes.",
    "args": [
    "UPGRADE_EXPORT_TIMEOUT",
    "UPGRADE_IMPORT_TIMEOUT",
    "UPGRADE_POST_IMPORT_TIMEOUT"
    ],
    "localized": "To extend the default timeout, set environment variable UPGRADE_EXPORT_TIMEOUT for export upgrade phase, UPGRADE_IMPORT_TIMEOUT for import upgrade phase, and  UPGRADE_POST_IMPORT_TIMEOUT for import historical data phase. The timeout is in minutes."
    }
    },
    "start_time": "YYYY-MM-DDThh:mm:ss.000Z",
    "end_time": "YYYY-MM-DDThh:mm:ss.000Z"

Resolution

This issue can be resolved by resetting the Update Manager database. The instructions to do so can be found in KB 316581.