HOWTO: Diagnose performance issues with PGP Universal Server

Products

• Symantec Products

Issue/Introduction

 

Resolution

PGP Universal Server provides several tools that can assist with diagnosing performance issues via the command line. Some common symptoms of hardware-related performance issues include delays in email delivery, slow response times or timeouts when accessing the web console, and unexpected errors in various logs within PGP Universal Server.

 

 

top

 

The top utility allows you to see the current CPU load, RAM usage, swap usage, and running processes on your Universal Server. It will have output similar to the following:

 

top - 06:26:07 up 27 days, 18:38,  1 user,  load average: 0.15, 0.09, 0.08
Tasks: 332 total,   1 running, 331 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.5%us,  4.0%sy,  0.0%ni, 94.3%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   1035124k total,   841168k used,   193956k free,    71316k buffers
Swap:  2096472k total,    84960k used,  2011512k free,   217816k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23889 root      16   0  2752 1032  676 R  3.0  0.1   0:00.04 top
    1 root      18   0  2168  624  596 S  0.0  0.1   0:02.48 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.25 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.40 events/0
    6 root      19  -5     0    0    0 S  0.0  0.0   0:00.00 khelper
    7 root      17  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
   10 root      10  -5     0    0    0 S  0.0  0.0   0:10.34 kblockd/0
   11 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid
  173 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/0
  176 root      10  -5     0    0    0 S  0.0  0.0   0:00.02 khubd
  178 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 kseriod
  240 root      19   0     0    0    0 S  0.0  0.0   0:00.01 khungtaskd
  242 root      16   0     0    0    0 S  0.0  0.0   3:22.49 pdflush
  243 root      10  -5     0    0    0 S  0.0  0.0   0:07.42 kswapd0
  244 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
  325 root      16   0  4916 1488 1236 S  0.0  0.1   0:00.09 bash
  462 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kpsmoused
  492 root      10  -5     0    0    0 S  0.0  0.0   0:00.27 mpt_poll_0
  493 root      13  -5     0    0    0 S  0.0  0.0   0:00.00 mpt/0
  494 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_0
  497 root      12  -5     0    0    0 S  0.0  0.0   0:00.00 ata/0

 

The CPU(s) line displays current CPU usage. The id field indicates how much of the CPU is idle. If this number is very low, it may mean you don't have enough CPU resources for PGP Universal Server. Upgrading the CPU or allocating additional CPU resources (for virtual machines) will help with this.

 

The Mem line displays current RAM usage. If the free field is low, it may indicate that you need to add or allocate additional memory to the server. Because PGP Universal Server uses PAE kernel extensions, it can support up to 64GB of RAM despite using a 32-bit version of Linux.

 

The Swap line displays current swap file usage in much the same way that RAM is displayed. If the Mem free is low and the Swap used is high, you will see a significant decrease in performance and should add or allocation additional RAM as soon as possible. Excessive swapping can also lead to system lockups as the system becomes non-responsive.

 

Beneath the swap line is a list of currently running processes. By deault, this will be sorted by CPU usage and will refresh about every 4 seconds. This can help identify which processes are using the most CPU time. By using the greater than (>) and less than (<) keys on your keyboard, you can change the order in which processes are sorted. From the default view, the greater than key will change the sort order to memory usage.

 

Once you are done reviewing the information in top, press q to quit the program.

 

sar

 

The sar utility will display history system usage statistics including CPU usage, RAM usage, and disk activity. This can help pinpoint specific bottlenecks in performance. PGP Universal Server retains data over a seven day period to help isolate potential hardware utilization issues.

 

To display CPU usage for the current day, run sar without any arguments. You will get output similar to the following:

 

Linux 2.6.18-194.26.1.el5PAE (keys.lab.jharris.local)               11/03/2011

12:00:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM       all      2.08      0.00      4.43      0.30      0.00     93.18
12:20:01 AM       all      1.45      0.00      3.80      0.03      0.00     94.72
12:30:01 AM       all      1.42      0.00      3.84      0.02      0.00     94.72
12:40:01 AM       all      1.45      0.00      3.99      0.03      0.00     94.54
12:50:02 AM       all      1.52      0.00      3.67      0.04      0.00     94.78
01:00:01 AM       all      1.53      0.00      3.84      0.02      0.00     94.62
01:10:01 AM       all      1.56      0.00      3.91      0.02      0.00     94.51
01:20:01 AM       all      1.57      0.00      3.90      0.04      0.00     94.49
01:30:02 AM       all      1.40      0.00      3.76      0.02      0.00     94.82
01:40:01 AM       all      1.88      0.00      4.44      0.04      0.00     93.64
01:50:01 AM       all      1.50      0.00      3.89      0.03      0.00     94.58
02:00:01 AM       all      1.46      0.00      3.71      0.03      0.00     94.80
02:10:01 AM       all      1.57      0.00      3.77      0.02      0.00     94.64
02:20:01 AM       all      1.55      0.00      3.84      0.02      0.00     94.59
02:30:01 AM       all      1.49      0.00      3.79      0.02      0.00     94.70
02:40:01 AM       all      1.48      0.00      3.69      0.02      0.00     94.81
02:50:01 AM       all      1.51      0.00      3.89      0.02      0.00     94.57
03:00:01 AM       all      1.45      0.00      3.84      0.02      0.00     94.68
03:10:01 AM       all      6.63      0.00      6.72      0.46      0.00     86.20
03:20:01 AM       all      1.52      0.00      5.00      0.02      0.00     93.47
03:30:02 AM       all      1.45      0.00      4.88      0.03      0.00     93.64
03:40:01 AM       all      1.35      0.00      4.94      0.02      0.00     93.68
03:50:01 AM       all      1.47      0.00      5.05      0.02      0.00     93.46
04:00:01 AM       all      1.43      0.00      4.90      0.02      0.00     93.65
04:10:01 AM       all      1.64      0.00      5.30      1.28      0.00     91.78
04:20:01 AM       all      1.46      0.00      4.72      0.02      0.00     93.80
04:30:01 AM       all      1.41      0.00      4.82      0.03      0.00     93.74
04:40:01 AM       all      1.51      0.00      4.90      0.03      0.00     93.56
04:50:02 AM       all      1.36      0.00      4.79      0.02      0.00     93.83
05:00:01 AM       all      1.45      0.00      4.84      0.02      0.00     93.69
05:10:01 AM       all      1.48      0.00      4.88      0.03      0.00     93.62
05:20:01 AM       all      1.41      0.00      4.70      0.02      0.00     93.86
05:30:01 AM       all      1.53      0.00      4.74      0.02      0.00     93.72
05:40:01 AM       all      1.48      0.00      4.97      0.09      0.00     93.46
05:50:01 AM       all      1.51      0.00      4.90      0.03      0.00     93.56
06:00:01 AM       all      1.42      0.00      4.67      0.03      0.00     93.88
06:10:01 AM       all      1.47      0.00      4.92      0.04      0.00     93.57
06:20:01 AM       all      1.55      0.00      4.68      0.03      0.00     93.74
06:30:02 AM       all      1.45      0.00      4.69      0.03      0.00     93.83
06:40:01 AM       all      1.71      0.00      5.07      0.02      0.00     93.20
06:50:01 AM       all      1.74      0.00      5.06      0.02      0.00     93.18
07:00:02 AM       all      1.46      0.00      4.56      0.03      0.00     93.95
07:10:02 AM       all      1.73      0.00      4.71      0.05      0.00     93.50
Average:          all      1.64      0.00      4.50      0.07      0.00     93.79

 

This displays the CPU usage every 10 minutes for the current day. The idle column shows how much CPU is unused. A low number could indicate that the CPU is not powerful enough to process all of the requests coming to the server. The user and system columns show how much CPU is being used by programs and the OS, respectively.

 

The iowait column indicates how much CPU time was spent waiting for requested data to be fetched from the hard drive. A high number can indicate that the hard drive is too slow. This can be resolved either with a faster hard drive or by building a RAID array to increase performance. RAID 0, 5, 10, and 50 all provide increased disk performance. It may also indicate a problem with drive health such as bad sectors or a failing drive. If this is the case, the dmesg log (/var/log/dmesg) may show error messages related to the drive. You may also see messages about running a file system check when the server is booting up.

 

You can use sar to help identify these kinds of disk bottlenecks. Run sar -W to get a list of disk swapping activity. Using sar -b will show how many transactions per second are hitting the hard drive. These can both help isolate swapping and drive performance issues, including if issues happen during a certain peak time.

 

You can also use sar to pinpoint memory or swap utilization issues. Run sar -r and you will get output similar to the following:

 

Linux 2.6.18-194.26.1.el5PAE (keys.lab.jharris.local)        11/03/2011

 

12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad

12:10:01 AM    143368    891756     86.15    102852    231664   2014236     82236      3.92     18760

12:20:01 AM    143740    891384     86.11    103128    231792   2014236     82236      3.92     18760

12:30:01 AM    142624    892500     86.22    103424    231840   2014236     82236      3.92     18760

12:40:01 AM    142624    892500     86.22    103756    231924   2014236     82236      3.92     18760

12:50:02 AM    142376    892748     86.25    103960    232056   2014236     82236      3.92     18760

01:00:01 AM    138904    896220     86.58    104232    232104   2014236     82236      3.92     18760

01:10:01 AM    141384    893740     86.34    104512    232252   2014236     82236      3.92     18760

01:20:01 AM    140640    894484     86.41    104784    232412   2014236     82236      3.92     18760

01:30:02 AM    135060    900064     86.95    105028    232464   2014236     82236      3.92     18760

01:40:01 AM    139100    896024     86.56    105304    232948   2014236     82236      3.92     18920

01:50:01 AM    138356    896768     86.63    105512    233084   2014236     82236      3.92     18920

02:00:01 AM    133148    901976     87.14    105756    233256   2014236     82236      3.92     19020

02:10:01 AM    138852    896272     86.59    106000    233368   2014236     82236      3.92     19020

02:20:01 AM    138604    896520     86.61    106184    233484   2014236     82236      3.92     19020

02:30:01 AM    136496    898628     86.81    106420    233516   2014236     82236      3.92     19020

02:40:01 AM    137984    897140     86.67    106640    233656   2014236     82236      3.92     19020

02:50:01 AM    137488    897636     86.72    106884    233764   2014236     82236      3.92     19020

03:00:01 AM    133644    901480     87.09    107068    233812   2014236     82236      3.92     19020

03:10:01 AM    231464    803660     77.64     54184    196256   2011508     84964      4.05     19172

03:20:01 AM    231232    803892     77.66     54756    196584   2011508     84964      4.05     19172

03:30:02 AM    229504    805620     77.83     55324    196608   2011508     84964      4.05     19172

03:40:01 AM    229820    805304     77.80     55860    196736   2011508     84964      4.05     19172

03:50:01 AM    228948    806176     77.88     56440    196832   2011508     84964      4.05     19172

04:00:01 AM    226884    808240     78.08     56968    196880   2011508     84964      4.05     19172

04:10:01 AM    203340    831784     80.36     64196    214652   2011508     84964      4.05     19172

04:20:01 AM    201976    833148     80.49     64728    214760   2011508     84964      4.05     19172

04:30:01 AM    202224    832900     80.46     65288    214784   2011508     84964      4.05     19172

04:40:01 AM    200860    834264     80.60     65756    214904   2011508     84964      4.05     19172

04:50:02 AM    201108    834016     80.57     66264    215000   2011508     84964      4.05     19172

05:00:01 AM    197016    838108     80.97     66848    215052   2011508     84964      4.05     19172

05:10:01 AM    198256    836868     80.85     67360    215172   2011508     84964      4.05     19172

05:20:01 AM    199496    835628     80.73     67868    215272   2011508     84964      4.05     19172

05:30:01 AM    199000    836124     80.78     68372    215304   2011508     84964      4.05     19172

05:40:01 AM    196516    838608     81.02     68968    216768   2011508     84964      4.05     19172

05:50:01 AM    195772    839352     81.09     69428    216932   2011512     84960      4.05     19200

06:00:01 AM    190812    844312     81.57     69960    216960   2011512     84960      4.05     19200

06:10:01 AM    195100    840024     81.15     70488    217348   2011512     84960      4.05     19200

06:20:01 AM    194604    840520     81.20     70976    217444   2011512     84960      4.05     19200

06:30:02 AM    190772    844352     81.57     71520    217832   2011512     84960      4.05     19200

06:40:01 AM    192384    842740     81.41     72068    217964   2011512     84960      4.05     19200

06:50:01 AM    191020    844104     81.55     72636    218052   2011512     84960      4.05     19200

07:00:02 AM    188644    846480     81.78     73084    218140   2011512     84960      4.05     19200

07:10:02 AM    185420    849704     82.09     73632    220064   2011512     84960      4.05     19200

Average:       176897    858227     82.91     82196    220505   2012651     83821      4.00     19055

 

The columns are, in order, RAM free, RAM used, percentage of RAM used, RAM used for buffers, RAM used for caching, swap free, swap used, percentage of swap used, and swap cached. A high %memused and/or %swpused indicate that you need more RAM to improve performance.

 

df

 

The df utility is used to show how much drive space is available and how much of it is being used. Disks with low free space may perform worse as file fragmentation will often be higher. Here is a sample of output using df -h:

 

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2              37G  3.5G   32G  11% /

/dev/sda1              99M   17M   77M  18% /boot

tmpfs                 506M  8.0K  506M   1% /dev/shm

 

In this example, the main partition on the hard drive, /dev/sda2, is 37GB with 11% in use. The -h argument shows the output in a friendly human-readable format. If you are running into disk space issues, you should consider upgrading to a larger hard drive or expanding your RAID array, if you have one.