Log forwarding to secondary syslog targets fails when a primary TCP target is unreachable

Products

• VMware vCenter Server

Issue/Introduction

 

• Secondary UDP syslog targets stop receiving logs if one TCP target is down.

• When the vCenter Server Appliance (VCSA) is configured with multiple remote syslog targets, specifically using the TCP protocol for at least one target, a "Head-of-Line Blocking" (HoLB) condition can occur.

• If a TCP target becomes unreachable (e.g., network latency, firewall drop, or server hang), the rsyslog worker thread remains occupied waiting for the TCP timeout/handshake. This blocks the processing of all other log messages, including those destined for healthy secondary targets (e.g., UDP listeners), eventually leading to queue saturation and log data loss.

• When the TCP syslog server is hung or unresponsive, the journalctl logs report the following messages:
  Command to view the logs : "journalctl -r"
  

  YYYY-MM-DDTHH:MM:SS vcsa fqdn rsyslogd[#####]: omfwd: remote server at tcp.server.ip:51428 seems to have closed connection. This often happens when the remote peer (or an interim system like a load balancer or firewall) shuts dow>
YYYY-MM-DDTHH:MM:SS vcsa fqdn rsyslogd[[#####]: action 'action-0-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2306.0 try https://www.r>
YYYY-MM-DDTHH:MM:SS vcsa fqdn rsyslogd[[#####]: action 'action-0-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2306.0 try https://www.rsyslog.com/e/2359 ]

   

 

Environment

VMware vCenter Server 

Cause

The rsyslog daemon processes remote destinations sequentially using a single worker thread. A connection failure or timeout on a TCP syslog target causes the thread to stall.
This synchronous bottleneck prevents the daemon from processing subsequent messages, effectively blocking delivery to all other healthy targets (including UDP) and eventually leading to queue saturation and log data missing.

The stated workflow is the default behavior of rsyslog.

Resolution

Broadcom Engineering is currently aware of the issue and working towards the fix in a future release.

Workaround:

1. Identify and Remove Unreachable Targets:

   • Log into the vCenter Server Appliance Management Interface (VAMI) at https://<vCenter-IP>:5480.

   • Navigate to Syslog > Forwarding Configuration.

   • Locate the unreachable TCP target and click Edit or Delete to remove it.

2. Use UDP where Possible:

   • Use the UDP protocol, which is connectionless and does not cause blocking in the same manner as TCP.