"Connection reset" exception when implementing .NET Native Client service running under IIS in GemFire

Products

• VMware Tanzu Gemfire

Issue/Introduction

When implementing .NET Native Client service running under IIS to allow thousands of users to access concurrently, you may encounter performance issues.

This occurs when there are a large number of connections, reset messages, and high CPU consumption on the GemFire cluster side. 

This is caused by the caching module not being designed and called with the proper pattern.

For example:

You may see "Connection reset" and "Terminated client" messages in the server logs and high CPU usage on GemFire Cluster members when thousands of users access .NET Native Client service concurrently, if the client service creates a new instance for each user access:

[warn 2022/01/09 15:52:14.566 CST server1 <ServerConnection on port 41005 Thread 41509> tid=0xbb07] Server connection from [identity(192.168.1.51(default_GeodeDS:140588:loner):32048:Native_jVyrzqGNFs140588:default_GeodeDS,connection=1; port=52282]: Unexpected IOException: 
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:210)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at org.apache.geode.internal.cache.tier.sockets.Message.fetchHeader(Message.java:830)
    at org.apache.geode.internal.cache.tier.sockets.Message.readHeaderAndBody(Message.java:677)
    at org.apache.geode.internal.cache.tier.sockets.Message.receive(Message.java:1154)
    at org.apache.geode.internal.cache.tier.sockets.Message.receive(Message.java:1167)
    at org.apache.geode.internal.cache.tier.sockets.BaseCommand.readRequest(BaseCommand.java:862)
    at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMessage(ServerConnection.java:774)
    at org.apache.geode.internal.cache.tier.sockets.OriginalServerConnection.doOneMessage(OriginalServerConnection.java:72)
    at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1212)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl.lambda$initializeServerConnectionThreadPool$3(AcceptorImpl.java:676)
    at org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
    at java.lang.Thread.run(Thread.java:748)

[warn 2022/01/09 15:52:14.566 CST server1 <ServerConnection on port 41005 Thread 41509> tid=0xbb07] ClientHealthMonitor: Unregistering client with member id identity(192.168.1.51(default_GeodeDS:140588:loner):32048:Native_jVyrzqGNFs140588:default_GeodeDS,connection=1 due to: Connection reset

[warn 2022/01/09 15:52:14.588 CST server1 <ClientHealthMonitor Thread> tid=0x13ff] ClientHealthMonitor: Unregistering client with member id identity(192.168.1.51(default_GeodeDS:140588:loner):32153:Native_fyo5vIvVS2140588:default_GeodeDS,connection=1 due to: Unknown reason

[warn 2022/01/09 15:52:14.588 CST server1 <ClientHealthMonitor Thread> tid=0x13ff] Monitoring client with member id identity(192.168.1.51(default_GeodeDS:140588:loner):32153:Native_fyo5vIvVS2140588:default_GeodeDS,connection=1. It had been 60247 ms since the latest heartbeat. Max interval is 60000. Terminated client.

Environment

Product Version: 10.2

Resolution

The SocketException occurs on the GemFire cache server side when the .Net Native Client service closes the socket connection before the response can be returned over the socket. A Connection reset means that a TCP RST was received. A TCP RST packet indicates that the remote end (In this case, the .Net Native Client service) is replying that the connection on which the previous TCP packet was sent is not recognized, maybe the connection has been closed, maybe the port is not open, etc.. 

In the above example, the .NET Native Client service created and closed a new native client instance for each user access, or per operation. With thousands of concurrent users or operations, the cache servers may not be able to keep up with the churn.

In order to avoid this kind of situation, we can implement the cache module as a common service so that the client requests can share a single native client instance.

Brief steps to implement .NET Native Client service running under IIS(C#):

1. Initialize GemFire cache from Application_Start() method in global.asax.cs.

namespace CacheSessionStateExample
{
    public class MVCSampleApplication : System.Web.HttpApplication
    {
        static Cache cache;
        protected void Application_Start()
        {           
            ......
            FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
            RouteConfig.RegisterRoutes(RouteTable.Routes);
            BundleConfig.RegisterBundles(BundleTable.Bundles);
			cache = BuildGemfireCacheByProperties();         
        }
        private static Cache BuildGemfireCacheProperties()
        {
            string path = "xxxxx";
            string XMLPath = "yyyyy";
            string logPath = "zzzzz";
           
            CacheFactory cacheFactory;
            try
            {
                Properties<string, string> properties = new Properties<string, string>();
                properties.Insert("log-file", logPath);
                //properties.Insert("statistic-sampling-enabled", "true");
                // properties.Insert("statistic-archive-file", "xxxxxxx");               
                cacheFactory = CacheFactory.CreateCacheFactory(properties);
            }
            catch (Exception ex)
            {
                Log.Error("Unable to set Gemfire properties with error " + ex.Message);
                throw ex;
            }
            return cacheFactory.Create();
        }
  }
}

2. Create a main GemFire Cache implementation class that includes region access methods.

namespace CacheSessionStateExample
{
    public class GemFireCache : ObjectCache
    {
        static Cache cache;
        IRegion<string, object> region;
        string regionName;
        private static Object cacheLock = new Object();

        public GemFireCache()
        {
            cache = CacheFactory.GetAnyInstance();
            this.regionName = "exampleRegion";
            region = setRegionName(this.regionName);
       }

        private IRegion<string, object> setRegionName(string regionName)
        {
            IRegion<string, object> region = cache.GetRegion<string, object>(regionName);
            if (region == null)
            {
                throw new ApplicationException("Could not find a region: '" + regionName + "' from the Gemfire cluster, or the connection to the server was lost.");
            }
            return region;
        }
    ......
	}
}

3. You can then call the cache operations from your service logic, such as MVC Controller.