vCenter Server crash and performance metrics stop working on the vSphere Client.
search cancel

vCenter Server crash and performance metrics stop working on the vSphere Client.

book

Article ID: 395132

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vCenter is crashing and access to vSphere Client UI is lost with an error:

    You have no privileges to view this object or it is deleted.



  • The issue keeps happening even after restarting the appliance in the course of few hours or days depending on the memory allocated to the vCenter.

  • Prior to the crash performance metrics stop working on the vSphere Client.
  • The vCenter Data collection levels for performance monitoring are set to Level 2
  • vCenter logs indicate that the vpxd service crashed due to memory exceeding hard limits:

    /var/log/vmware/vpxd/vpxd.log
    xxxx-xx-xxTxx:xx:xx.xxxZ error vpxd[11045] [Originator@6876 sub=Memory checker] Current value xxxxxxxx exceeds hard limit xxxxxxxx. Shutting down process.
    xxxx-xx-xxTxx:xx:xx.xxxZ  panic vpxd[11045] [Originator@6876 sub=Default]
    -->
    --> Panic: Memory exceeds hard limit. Panic
    --> Backtrace:
    --> [backtrace begin] product: VMware VirtualCenter, version: x.x.x, build: build-xxxxxxxx, tag: vpxd, cpu: x86_64, os: linux, buildType: release
    --> backtrace[00] libvmacore.so[0x00531DC5]
    --> backtrace[01] libvmacore.so[0x0042182A]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
    --> backtrace[02] libvmacore.so[0x00434009]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
    --> backtrace[03] libvmacore.so[0x0050A989]
    --> backtrace[04] libvmacore.so[0x0050AAA1]: Vmacore::PanicExit(char const*)
    --> backtrace[05] libvmacore.so[0x0042154C]: Vmacore::System::ResourceChecker::DoCheck()
    --> backtrace[06] libvmacore.so[0x00385107]
    --> backtrace[07] libvmacore.so[0x0037EC04]
    --> backtrace[08] libvmacore.so[0x00384517]
    --> backtrace[09] libvmacore.so[0x00510FBB]
    --> backtrace[10] libpthread.so.0[0x00008EB0]
    --> backtrace[11] libc.so.6[0x000FFADF]
    --> backtrace[12] (no module)
    --> [backtrace end]

Environment

VMware vCenter 7.x
VMware vCenter 8.x

Cause

Issue is related to stale index statistics residing in vCenter's vPostgres database

Resolution

To address the issue clean the stale statistics using steps from KB article 313506


Command to be used:

 /opt/vmware/vpostgres/current/bin/psql -U postgres -v TaskMaxAgeInDays=-1 -v EventMaxAgeInDays=-1 -v StatMaxAgeInDays=30 -d VCDB -t -q -f /root/2110031_Postgres_task_event_stat_reset_event_sequence.sql

The above will keep the tasks and events unchanged and will clean up statistics retaining only the last 30 days (default retention policy)

Additional Information

Powered by Wolken Software