Lifecycle manager page shows an error "An unexpected error has occurred"

Products

• VMware vCenter Server

Issue/Introduction

• Lifecycle manager page showing  "An unexpected error has occurred"

 

• vCenter > Updates > Baseline page showing "An unexpected error has occurred"

 

• vCenter > Updates > Image page showing "no healthy upstream"

 

• The vSphere Update Manager service fails to start.

user@vcenter-app[ /usr/lib/vmware-updatemgr/bin ]# service-control --stop vmware-updatemgr && service-control --start vmware-updatemgr

Operation not cancellable. Please wait for it to finish...
Performing stop operation on service updatemgr...
Successfully stopped service updatemgr
Operation not cancellable. Please wait for it to finish...
Performing start operation on service updatemgr...
Error executing start on service updatemgr. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "updatemgr"
            ],
            "localized": "An error occurred while starting service 'updatemgr'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "updatemgr"
            ],
            "localized": "An error occurred while starting service 'updatemgr'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

 

• In vCenter, /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log - "Failed to parse /usr/lib/vmware-updatemgr/bin/jetty-vum-ssl.xml.  Error: no element found"

YYYY-MM-DDThh:mm:ss.429Z info vmware-vum-server[1773811] [Originator@6876 sub=managedJetty] [managedJetty 212] commandout YYYY-MM-DD hh:mm:ss.425:INFO::main: Logging initialized @149ms to org.eclipse.jetty.util.log.StdErrLog
-->
YYYY-MM-DDThh:mm:ss.429Z info vmware-vum-server[1773811] [Originator@6876 sub=managedJetty] [managedJetty 212] commandout rdlitzf
-->
YYYY-MM-DDThh:mm:ss.429Z info vmware-vum-server[1773811] [Originator@6876 sub=managedJetty] [managedJetty 212] commandout OBF:1vuz1x0p1xfb1unz1xfr1x1x1vub
-->
YYYY-MM-DDThh:mm:ss.432Z error vmware-vum-server[1773811] [Originator@6876 sub=managedJetty] [managedJetty 328] Failed to parse /usr/lib/vmware-updatemgr/bin/jetty-vum-ssl.xml.  Error: no element found
YYYY-MM-DDThh:mm:ss.434Z error vmware-vum-server[1773811] [Originator@6876 sub=managedJetty] [managedJetty 328] [backtrace begin] product: VMware Update Manager, version: 8.0.3, build: build-24853646, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x00531DD5]
--> backtrace[01] libvmacore.so[0x00421834]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x00434013]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvci-vcIntegrity.so[0x007C3B5A]: Integrity::ManagedJetty::UpdateConfigFilePassword(Sysimage::ZeroingString const&)
--> backtrace[04] libvci-vcIntegrity.so[0x007C77EF]: Integrity::ManagedJetty::Start()
--> backtrace[05] libvci-vcIntegrity.so[0x0070729D]: Integrity::VcIntegrityImpl::GetManagedProcesses()
--> backtrace[06] libvci-vcIntegrity.so[0x00707F79]: Integrity::VcIntegrityImpl::VcIntegrityImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
--> backtrace[07] libvci-vcIntegrity.so[0x0070822C]: Integrity::CreateVcIntegrity(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
--> backtrace[08] libvci-vcIntegrity.so[0x006F9003]: Integrity::VcIntegrityPlugin::Init()
--> backtrace[09] libvmacore.so[0x004B8A47]
--> backtrace[10] updatemgr[0x0000E6F0]
--> backtrace[11] updatemgr[0x0000EF9A]
--> backtrace[12] updatemgr[0x0000D678]
--> backtrace[13] updatemgr[0x0000FF68]
--> backtrace[14] libc.so.6[0x00027F0A]
--> backtrace[15] updatemgr[0x0000D029]
--> backtrace[16] (no module)
--> [backtrace end]
YYYY-MM-DDThh:mm:ss.468Z warning vmware-vum-server[1773937] [Originator@6876 sub=SysCommandPosix] Status of pid ####### is discarded

Environment

• vCenter 6.x
• vCenter 7.x
• vCenter 8.x
• vCenter 9.x

Cause

This issue occurs if the jetty-vum-ssl.xml file is missing or corrupt.

Resolution

Use one of the below options to resolve

Option#1

• Recover the jetty-vum-ssl.xml from vCenter backup
  • <Backupserver>\<BackupFolder>\vCenter\sn_vcenter_fqdn\M_<vCenter-Version>_<Date>-<Time>_\config_files.tar.gz
  • Extract config_files.tar.gz file
  • config_files\usr\lib\vmware-updatemgr\bin
  • Find the file "jetty-vum-ssl.xml"
• Take Winscp to the vCenter server
• Upload and ovrewrite the "jetty-vum-ssl.xml" recovered from backup upload into vCenter this location - "/usr/lib/vmware-updatemgr/bin/"
• Restart the services -  service-control --stop vmware-updatemgr && service-control --start vmware-updatemgr

Option#2

• If the vCenter is on ELM or other working vCenter.
• Copy the /usr/lib/vmware-updatemgr/bin/jetty-vum-ssl.xml file
• Upload into the broken update manager vCenter server
• Restart the services -  service-control --stop vmware-updatemgr && service-control --start vmware-updatemgr