High CPU on LSASS within Windows Guest OS due to excessive VIX API authentication attempts

Products

• VMware vSphere ESXi

Issue/Introduction

• A Windows virtual machine may exhibit high CPU usage in the LSASS.exe process. This is accompanied by a massive volume of authentication events for a specific service or domain account. Also, this is observed during backup windows.
• The following logs are observed in the ESXi path /var/log/hostd.log:

  <timestamp> Db(167) Hostd[2099822]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/datastore/vm-folder/vm-name.vmx] Change file attributes translated error to vmodl.fault.InvalidArgument
  <timestamp> Db(167) Hostd[2099822]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/datastore/vm-folder/vm-name.vmx] Change file attributes message:
  <timestamp> In(166) Hostd[2099827]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 14603 : Guest operation Change File Attributes performed on Virtual machine vm-name.
  <timestamp> In(166) Hostd[2099827]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/datastore/vm-folder/vm-name.vmx] State Transition (VM_STATE_GUEST_OPERATION -> VM_STATE_ON)
  <timestamp> In(166) Hostd[2099827]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/datastore/vm-folder/vm-name.vmx] [N8Guestsvc31ChangeFileAttributesRequestImplE:0x00000096d786ee70] opCode=14 auth=<hidden> guestFilePath=C:\Windows\TEMP\VixProxy_-id permissions=484 failed
  <timestamp> In(166) Hostd[2099827]: [Originator@6876 sub=Solo.Vmomi] Activation finished; <<#####-####-####-####-##########, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 40965'>>, ha-guest-operations-file-manager, vim.vm.guest.FileManager.changeFileAttributes, <vim.version.v8_0_3_0, internal, 8.0.3.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x00000096d7d96db8]>
  <timestamp> Db(167) Hostd[2099827]: [Originator@6876 sub=Solo.Vmomi] Arg vm:
  <timestamp> Db(167) Hostd[2099799]: --> 'vim.VirtualMachine:##'
  <timestamp> Db(167) Hostd[2099827]: [Originator@6876 sub=Solo.Vmomi] Arg auth:
  <timestamp> Db(167) Hostd[2099799]: --> (vim.vm.guest.NamePasswordAuthentication) {
  <timestamp> Db(167) Hostd[2099799]: -->    interactiveSession = false,
  <timestamp> Db(167) Hostd[2099799]: -->    username = "Domain\User",
  <timestamp> Db(167) Hostd[2099799]: -->    password = (not shown)
  <timestamp> Db(167) Hostd[2099799]: --> }
  <timestamp> Db(167) Hostd[2099827]: [Originator@6876 sub=Solo.Vmomi] Arg guestFilePath:
  <timestamp> Db(167) Hostd[2099799]: --> "C:\Windows\TEMP\VixProxy_-id"
  <timestamp> Db(167) Hostd[2099827]: [Originator@6876 sub=Solo.Vmomi] Arg fileAttributes:
  <timestamp> Db(167) Hostd[2099799]: --> (vim.vm.guest.FileManager.PosixFileAttributes) {
  <timestamp> Db(167) Hostd[2099799]: -->    permissions = 484,
  <timestamp> Db(167) Hostd[2099799]: --> }
  <timestamp> In(166) Hostd[2099827]: [Originator@6876 sub=Solo.Vmomi] Throw vmodl.fault.InvalidArgument
  <timestamp> In(166) Hostd[2099827]: [Originator@6876 sub=Solo.Vmomi] Result:
  <timestamp> In(166) Hostd[2099799]: --> (vmodl.fault.InvalidArgument) {
  <timestamp> In(166) Hostd[2099799]: -->    msg = "",
  <timestamp> In(166) Hostd[2099799]: --> }

Cause

The issue is caused by external applications like backup software repeatedly calling Guest Operations APIs with invalid arguments. In this specific scenario, the application attempts to use PosixFileAttributes on a Windows file system via the changeFileAttributes method. This results in an vmodl.fault.InvalidArgument error and triggers continuous re-authentication attempts, which exhausts LSASS resources.

Resolution

To resolve this issue, address the configuration of the third-party application triggering the VIX API calls with an invalid argument.