Excessive logging consuming large space in syslog server during VM backup window

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Syslog server receives a very high number of events from the ESXi server during the backup window
• Sample events from vpxa or hostd log indicate NFC debug enabled logs
  ####-##-##T##:##:##.###Z info vpxa[#######] [Originator@6876 sub=Libs] [NFC DEBUG] NfcAioRecvData: Srv receiving data, size = 44
####-##-##T##:##:##.###Z info vpxa[#######] [Originator@6876 sub=Libs] [NFC DEBUG] NfcAioServerProcessMain: Processing an operation
####-##-##T##:##:##.###Z info vpxa[#######] [Originator@6876 sub=Libs] [NFC DEBUG] NfcAioIoCB: Received callback
####-##-##T##:##:##.###Z info vpxa[#######] [Originator@6876 sub=Libs] [NFC DEBUG] NfcAioServerProcessMain: Processing an operation
####-##-##T##:##:##.###Z info vpxa[#######] [Originator@6876 sub=Libs] [NFC DEBUG] NfcAioIoCB: Received callback
####-##-##T##:##:##.###Z info vpxa[#######] [Originator@6876 sub=Libs] [NFC DEBUG] NfcAioServerProcessMain: Processing an IO callback
####-##-##T##:##:##.###Z info vpxa[#######] [Originator@6876 sub=Libs] [NFC DEBUG] NfcAioSendMessage: Srv sending msg, opId = #######, type = 7, size = 44

• The /var/run/log/vmauthd.log on ESXi host points at the server requesting debug logging
  ####-##-##T##:##:##.###Z vmauthd[########]: Connect from remote socket (<IP Address>:32938).
####-##-##T##:##:##.###Z vmauthd[########]: Connect from <IP Address>
####-##-##T##:##:##.###Z vmauthd[########]: Received PROXY command for vpxa-nfc, session = ## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##
####-##-##T##:##:##.###Z vmauthd[########]: vmware-vpxa: Connected to vpxa-nfc-fd, remote end sent pid: #######

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

The issue is caused by the configuration in the Backup application requesting debug logging which is causing the log spew. 

The steps may vary from Backup vendor to vendor, so consult their Backup Team/Vendor for precise steps. 

Resolution

In case of Veritas, the following file 'vixDiskLib.ini' is located at /usr/openv/lib/shared/vddk/lib64/

Sample file 

vixDiskLib.disklib.EnableCache = 0
vixDiskLib.nfc.LogLevel = 4
vixDiskLib.transport.LogLevel = 6
vixDiskLib.nfcFssrvr.timeoutMs = 0

The vixDiskLib.nfc.LogLevel needs to be set to 1 (error only) which should ensure the log spew is avoided. No service restart is needed on ESXi host, but check with the backup Vendor if the application would require restart for changes to be effective. 

Additional Information

About the vixDiskLib.nfc.LogLevel option:

• Initialize Virtual Disk API
• Virtual Disk Development Kit 8.0 Update 2 Release Notes - How to Change NFC Logging Level

 

仮想マシンバックアップ中の過剰なロギングが syslog サーバーの領域を大量に消費する