Login Events “logged in / logged out” messages are recorded every five minutes in hostd logs on a HPE Host.

Products

• VMware vSphere ESXi

Issue/Introduction

1. Per hostd.log, the following login event in seen every 4-5 minutes trying to run an "esxcli" command. For example, in the scenario below, you see the user root login to run the esxcli command to get the system maintenance mode information.

Event 884 : User [email protected] logged out (login time:, number of API invocations:, user agent: pyvmomi Python/3.8.18 (VMkernel; 8.0.2; x86_64))
In() Hostd[]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-<op-ID> sid=<id> user=root] Dispatch system.maintenanceMode.get
In() Hostd[]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-<op-ID> sid=<id> user=root] Dispatch system.maintenanceMode.get done

 

2. On running either of the below command on the ESXi shell to identify the parent process responsible for generating the esxcli command, you see the parent process belonging to "SUT". 

while true; do (ps -CcJ | grep -v grep | grep -B 5 -A 1 /bin/esxcli >> /tmp/ps_CcJ.txt); sleep 1; done
or
while true; do date >> /tmp/ps_CcJ.txt && ps -CcJ >> /tmp/ps_CcJ.txt; sleep 1; done

 

3. In the example below, it runs the command "esxcli system maintenanceMode get" and sends the command output to the sutmaintenancemode.log under /var/tmp/sut/

2101969  2101969  sut                   /opt/sut/bin/sut
7049220  7049220  sh                    sh -c esxcli system maintenanceMode get >& /var/tmp/sut/stagingdirectory/sutmaintenancemode.log
7049221  7049221  python                python /sbin/esxcli system maintenanceMode get

4. In case the instructions above are not able to confirm the process, the steps in kb- https://knowledge.broadcom.com/external/article/319996 under "B. For esxcli:" confirm the same.

Environment

VMware vSphere ESXi

VMware vCenter Server

Cause

The source of the incoming command is "SUT" (System Update Tool) which repeatedly logs in and logs out of the ESXi host via esxcli.

Resolution

System Update Tool (SUT) is an application that facilitates the deployment of firmware and software through the Integrated Lights-Out (iLO) management network using the server's local administrator credentials. Contact HPE for the resolution on this.

Workaround:

The below steps should only be performed after getting a confirmation from HPE.

1. To ascertain whether SUT is responsible for generating the events, please disable it on one of the hosts and monitoring if the events cease. To stop the service, please use the following commands:

    a. sut -stop
    b. sut -start

 

2. It is important to note that in some instances where similar behavior has been observed, merely disabling SUT has proven insufficient, therefore, if the events persist, we should consider removing it entirely. To remove SUT, please execute the following commands:

    a. sut -deregister
    b. esxcli software component remove -n sutComponent

3. Then reboot the ESXi Host.