Smarts SAM: Smarts Broker does not respond after bouncing services

Products

• VMware

Issue/Introduction

Symptoms:




Smarts Broker does not respond after bouncing services
After bouncing Smarts SAM and OI from primary to backup, the Smarts Broker stops replying and SAM and OI will not start

Executing Smarts brcontrol command does not show any result and consoles could not attach to SAM
 

Smarts SAM and OI logs contain errors similar to the following:

[24-ene-2011 16:42:45+030ms W. Europe Standard Time] t@1376 HTTP #2
CI2-W-CI_BLOCK_READ_TIMEOUT-Timeout exceeded for block read; in file
"h:/FOUNDATION-7.2.0.X/118/smarts/clsapi/ci_flow.c" at line 2731
[24-ene-2011 16:42:52+952ms W. Europe Standard Time] t@2864 Flow KeepAlives #2
CI-E-EFLOWID-For flow CI_FlowTCP_U [Queued HTTP client] IN_FLOW|PHYSICAL
@0x0358ed40
. *:v4:426 KS 24-ene-2011 16:41:53 W. Europe Standard Time, KR 24-ene-2011
16:42:13 W. Europe Standard Time F:0
. Open fd=12284, 0.0.0.0:426 -> 10.98.154.48:2139, tmo 0 00:00:15 N/S 1/2
CI-EFLOWKEEPALIVESEND-Flow closing due to inability to send keepalive
<SYS>-zWSAECONNRESET-Connection reset by peer

 

Smarts logs contain one of the following error messages:

WSAETIMEDOUT-Connection timed out
WSAECONNRESET-Connection reset by peer

Environment

VMware Smart Assurance - SMARTS

Cause

This problem is caused by environments that use the Windows TCP Connection Open Rate Limitation. Some Service Packs of Windows limit the number of tcp connections that can be opened to 10/sec, in an attempt to harden the TCP/IP stack against denial of service attacks. This can cause problems in the normal working of Smarts.  See MSDN KB Article Q324270 for more information on this issue.

This limit manifests itself in Smarts logs typically as one of the following error messages:

WSAETIMEDOUT-Connection timed out
WSAECONNRESET-Connection reset by peer

Resolution

Ensure that the brokers are running with at least Foundation 7.2 build 148. For Windows environments, you can then try disabling the SynAttackProtect for the version of Windows used in your environment as described in the following sections.

Disabling SynAttackProtect in Win2k3 SP1/WinXP SP2 in the registry
To remove the limit introduced in SP1 and revert the limit back to its original setting (unlimited), do the following in the registry: 

1. 1. Add the following registry key with type DWORD and value of 0:
   
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\SynAttackProtect
   
    
2. Shutdown all servers on the system and restart the entire system
3. Start Smarts Broker and restart other servers that were shutdown in preceding step.

Disabling SynAttackProtect in Windows Server 2008 / Windows Vista in the registry
By default, the half-open TCP connections limit is disabled in Windows Server 2008 with Service Pack 2 (SP2) and in Windows Vista with Service Pack 2 (SP2). This article describes how to impose the half-open TCP connections limit in Windows Server 2008 with SP2 and in Windows Vista with SP2. The limit is ten connections. Note In Windows Server 2008 and in Windows Vista with Service Pack 1 (SP1), the system allows for a maximum of ten half-open TCP connections at any time.

To disable SynAttackProtect in Windows Server 2008 / Windows Vista environments, do the following:

1. Locate the following registry key and set its value as 0:
   
      HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableConnectionRateLimiting
   
   
   Note: See Microsoft KB Article 969710 for information on the Registry key in Windows Vista.
   
    
2. Shutdown all servers on the system.
3. Restart the system.
4. Start Smarts Brokerand restart other servers that were shutdown in preceding step.