Gemfire: Observed performance difference between Cisco and Dell servers

Products

• VMware Tanzu Gemfire

Issue/Introduction

GemFire clusters running on Cisco hardware see periodic client latency, higher client connection churn, and spikes in disk store queues compared to those on Dell servers. This occurs even with identical client pool and GemFire server configurations.On the hardware side, the difference is that Cisco uses a Cisco network adapter and SAN while Dell servers do not. What is driving this discrepancy?

 

 

Environment

All supported Gemfire versions on Cisco and Dell hardware.

Cause

On a scheduled interval (e.g., every 5 minutes), Cisco UCS Manager (UCSM) polls low-level hardware components—such as Cisco VIC (Virtual Interface Card) network adapters, RAID controllers, and memory buses—to collect telemetry, temperature, and performance metrics. This frequent polling results in hardware interrupts which CPUs have to service. This temporary CPU overload/starvation can cause JVM threads to freeze which can impact Gemfire operations causing the observed behavior, like latency and churn. Since standalone Dell servers use local chipsets they do not have the same issue with hardware-to-controller polling and interrupts.

Additionally, identical security scanners on both hosts can impact Cisco servers more due to this unique hardware.

 

Resolution

Because hardware-level management and security scanners cannot be disabled, the application layer must be configured to tolerate infrastructure noise:

• Tune Client Connection Pools: Apply recommended connection pool configurations (min/max-connections, load-conditioning-interval, read timeouts) across all client applications to absorb host latency blips without dropping sockets.

•  Monitor client stats post-tuning to confirm connection churn is eliminated during background host spikes.