VMware vCenter Server database grows with the logging of [email protected] events

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• The vCenter Server database grows unexpectedly.
• The vCenter Server Tasks and Events tab frequently reports one of the following message:
  
  User [email protected] logged in
  
  User root logged out
  
  
• This issue affects database tables such as VPX_EVENT and VPX_EVENT_ARG.
• This issue occurs when ESXi hosts are running on HP Gen8 hardware.
• The hostd.log file (located in /var/log/) contains frequent entries similar to:
  
  info 'Vimsvc'] [Auth]: User root
  info 'ha-eventmgr'] Event 491 : User [email protected] logged in
  info 'ha-eventmgr'] Event 492 : User root logged out

Environment

VMware vCenter Server 5.1.x
VMware vSphere ESXi 5.0
VMware vCenter Server 5.0.x
VMware vSphere ESXi 5.1

Cause

This issue occurs when the hostd process in each ESXi host reports all logins made to the system. This is an expected behavior.

There are three components involved:

1. hostd: This sends an event for each login attempt.
   
   Note: The hostd process cannot be configured to stop or combine login events.
   
   
2. Application: This is installed on the ESXi host via the hostd process over TCP ports 80 and 443.
   
   Note: As these applications are all running on ESXi, they may access hostd more frequently than external applications, and therefore generate a large number of login events.
   
   
3. vCenter Server database: Events are collected and stored for each hostd agent sending events.
   
   Note: vCenter Server does not automatically combine login events. This can cause the events tables to grow and fill the database. An alarm or alert does not trigger in vCenter Server.

Resolution

To resolve this issue for HP Gen 8 servers, apply one of these bundles.

• HP ESXi Offline Bundle for VMware ESXi 5.0 and vSphere 5.1 - Version 1.5
• HP Agentless Management Service Offline Bundle for VMware ESXi 5.0 and vSphere 5.1 - Version 9.4.0

For more information, see the HP Customer Advisory article C03909206.

Note: The preceding links were correct as of December 18, 2013. If you find a link is broken, provide feedback and a VMware employee will update the link.

To work around this issue, stop the hp-ams daemon running on the ESXi host.

Note: These steps do not affect the vCenter Server's ability to monitor or report on the ESXi host hardware health. However, HP iL0 4 cannot monitor the ESXi hypervisor status after you apply these steps.

To stop the hp-ams daemon:

1. Log in to the ESXi host as the root user.
2. Verify that the ESXi host is an HP GEN8 system by running the command:
   
   esxcfg-info | head
   
   You see output similar to:
   
   \==+Hardware Info :
   |----BIOS UUID................................................0x30 0x31 0x34 0x36 0x36 0x31 0x53 0x55 0x45 0x32 0x34 0x36 0x4d 0x30 0x4a 0x4e
   |----Product Name.............................................ProLiant BL460c Gen8
   
   Note: Gen8 is noted in the Product Name field.
   
   
3. If the host is an HP Gen8 system, stop the hp-ams daemon by running the command:
   
   /etc/init.d/hp-ams.sh stop
   
   
4. To persist the change in the event of a host reboot, run the command:
   
   chkconfig hp-ams.sh off

An additional step may be required for ESXi 5.1 hosts. This step allows you to change the logging interval in the crontab file.

Note: The default logging interval is 5 minutes.

To change the logging interval in the crontab file:

1. On the ESXi 5.1 host, run the command:
   
   cat /var/spool/cron/crontabs/root | grep hostd-probe
   
   You see the output:
   
   */5 * * * * /sbin/hostd-probe
   
   Note: This indicates that the hostd-probe runs every 5 minutes.
   
   
2. Open the crontab file using a text editor.
   
3. Change the logging interval. For example, to set the logging interval to 50 minutes, change the line to:
   
   */50 * * * * /sbin/hostd-probe
   
   
4. Save and close the file.

Additional Information

ESXi ramdisk full due to /var/log/hpHelper.log file size