Message lost in vmsyslog occur on ESXi hosts in an NSX DFW environment
search cancel

Message lost in vmsyslog occur on ESXi hosts in an NSX DFW environment

book

Article ID: 417805

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • In /var/log/.vmsyslogd.err see entries similar to:

    YYYY-MM-DDThh:mm:ss.###Z vmsyslog.loggers.network : ERROR   ] <Syslog Server IP address>:514 - socket error : timed out
    YYYY-MM-DDThh:mm:ss.###Z vmsyslog.loggers.network : ERROR   ] <Syslog Server IP address>:514 - socket init calls failed: <class 'OSError'> 
    YYYY-MM-DDThh:mm:ss.###Z vmsyslog.loggers.network : ERROR   ] failed to establish connection to remote syslog server <Syslog Server IP address>:514
    YYYY-MM-DDThh:mm:ss.###Z vmsyslog.msgQueue        : ERROR   ] <Syslog Server IP address>:514 - lost ##### log messages

  • FIREWALL-PKTLOG entries are being written to /var/run/log/dfwpktlogs.log on the ESXi host at an extremely high rate, such as thousands to tens of thousands of lines per second.

    Example Output:

    YYYY-MM-DDThh:mm:ss.###Z No(##) FIREWALL-PKTLOG[#####]: ######## INET match PASS #### OUT ## TCP ###.###.###.###/###->###.###.###.###/### S <Rule Name>
    YYYY-MM-DDThh:mm:ss.###Z No(##) FIREWALL-PKTLOG[#####]: ######## INET match PASS #### OUT ## TCP ###.###.###.###/###->###.###.###.###/### S <Rule Name>
    YYYY-MM-DDThh:mm:ss.###Z No(##) FIREWALL-PKTLOG[#####]: ######## INET match PASS #### OUT ## TCP ###.###.###.###/###->###.###.###.###/### S <Rule Name>
    YYYY-MM-DDThh:mm:ss.###Z No(##) FIREWALL-PKTLOG[#####]: ######## INET match PASS #### OUT ## TCP ###.###.###.###/###->###.###.###.###/### S <Rule Name>
    YYYY-MM-DDThh:mm:ss.###Z No(##) FIREWALL-PKTLOG[#####]: ######## INET match PASS #### OUT ## TCP ###.###.###.###/###->###.###.###.###/### S <Rule Name>

    Note:  The above message differs based on the packet log configuration of DFW.

Environment

VMware vSphere ESXi

Cause

Enabling packet logging in NSX DFW can cause a high volume of entries, potentially thousands to tens of thousands of lines per second, to be written to /var/run/log/dfwpktlogs.log on the ESXi host.
This impact may cause message lost in vmsyslog.

Resolution

Verify the packet logging settings on the NSX DFW. For more information, see Excessive Distributed Firewall (DFW) Logging Causes Host Resource or Stability Issues.

Additional Information

Powered by Wolken Software