Moving the DB to a Solaris environment w/ highly threaded CPU architecture results in extreme slow performance

Products

• Data Loss Prevention Enforce

Issue/Introduction

Moving the DB to a Solaris environment results in extreme slow performance.

The specific hardware architecture is a highly threaded Sun CPU such as Ultra SPARC T1 , Ultra SPARC T2 , SPARC 64 VII , Ultra SPARC RK , SPARC VIIIfx , Niagara 3 .

Resolution

Problem:

Performance degradation is per Symantec's analysis caused by the hardware architecture that the Oracle DB environment is running on. The hardware is designed specifically to highly concurrent, small query size while the Symantec DLP application has by nature low query volume but large query data size.

Certified and fully support Oracle DB versions

 

Symantec DLP is certified for Oracle Database installations on Windows 2003 and Redhat Linux (32-bit only).  Support responsibility for Oracle installed on ANY OTHER platform falls to the customer.  The following excerpt is from the "Symantec Data Loss Prevention System Requirements and Compatibility Guide, version 10.0", page 20 :

 

Oracle database requirements

 

Symantec Data Loss Prevention requires the Oracle 10g database version 10.2.0.4

with the most recent Critical Patch Update. Oracle can be run on Windows Server

2003 (any version) and Red Hat Enterprise Linux (any version) operating systems.

 

See as reference the following public facing articles:

 

TECH221684  ( Which platforms does Symantec DLP require for Oracle installation? )

TECH220433 ( What would occur in current releases if the backend Oracle DB is not certified or unsupported ? )

 

 

Technical Details:

 

In general, the performance impacting factors are for the user experience are

- Web browser

- Network throughput

- App side performance (Enforce)

- DB performance

 

Verify that the network load, the browser CPU usage and the Enforce Server are not loaded when bad performance is observed.

This is to ensure that you are not running into multiple attributing factors. If this is the case, those need to be investigated separately.

 

Further investigation focused on the DB side. Enabling JDBC logging showed that the DB response is fairly slow.

JDBC Logging can be enabled via the steps outlined in TECH219627

 

CPU

 

Most modern Sun CPUs (Ultra SPARC T1, Ultra SPARC T2, SPARC 64 VII, Ultra SPARC RK, SPARC VIIIfx, Niagara 3)  are highly threaded and are tailored for web application usage which are highly threaded and therefore benefit such a processor. They are however not ideal for application server environments. Especially the T1 and T2 processors are available in a 1 GHz - 1.6 Ghz. So if you have an App Server you are better performing in a low core/thread environment due to the higher frequency and cache usage.

As an example, in a Ultra SPARC IV the 2 cores share 64 KB L1 DCache, 32 KB L1 lcache, 16 MB L2 cache  while in a Ultra SPARC T2 the 64 threads would share 8 KB L1 DCache, 16 KB L1 lcache, 4 MB L2 cache.

 

The T-Series architecture is designed for heavy multithreading, but for our intents and purposes unless we introduce high magnitude of multithreading and they are under high load.

 

Symantec DLP is in contrast an application which is characterized through:

- Low query volumes, low amount of queries

- Large individual query size

 

As a result we are the worst-case scenario for this environment.

Wiki reference to the T1 (http://en.wikipedia.org/wiki/UltraSPARC_T1 ) summarizes under 'Target market' the architectural disadvantage in more detail:

 

[...]the chip is targeted at network-facing high-demand servers, such as high-traffic web servers, and mid-tier Java, ERP, and CRM application servers, which often utilize a large number of separate threads. One of the limitations of the T1 design is that a single floating point unit (FPU) is shared between all 8 cores, making the T1 unsuitable for applications performing a lot of floating point mathematics.

 

Oracle discussed in their Metalink article 781763.1 (Migration from fast single threaded CPU machine to CMT UltraSPARC T1 and T2 results in increased CPU reporting) this very topic and compares the performance of a T2000 UltraSPARC machine vs. an older V440 SPARC machine. (Via Oracle Metalink or http://hemanshusharma.blogspot.com/2009/11/migration-from-ultrasparc-i-v-to.html )

 

They note:

 

While Sun may report that all the virtual CPU on a T2-based computer as being 1200MHz (1.2GHz) -- for example, when looking at "/usr/sbin/psrinfo -v" output, the work within each physical core is actually divided across 4 (Chip-level MultiThreading - or CMT) threads. So any one active Unix process will only use 25% of the CPU core's cycles (or effectively feel like it is on a single CPU that is only running at 300MHz).

               

So if they have a 1.6 Ghz machine and 8 threads per core, this will result in an individual thread running to some extend as a 200 Mhz per thread. In addition with the single FPU that is being used across the threads, it causes additional slowdown.

For how to gather environment comparative information see TECH220547. Recommendations

Based on the above the following may increase through put:

 

 

 

- Place the DB in standard supported environment

- Place DB on a non-threaded processor, you may consult with your Account Manager to request reference customers who use Oracle on Solaris for reference implementations