Resolution:
PGP Universal Server provides several tools that can assist with diagnosing performance issues via the command line. Some common symptoms of hardware-related performance issues include delays in email delivery, slow response times or timeouts when accessing the web console, and unexpected errors in various logs within PGP Universal Server.
top
The top utility allows you to see the current CPU load, RAM usage, swap usage, and running processes on your Universal Server. It will have output similar to the following:
top - 06:26:07 up 27 days, 18:38, 1 user, load average: 0.15, 0.09, 0.08
Tasks: 332 total, 1 running, 331 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.5%us, 4.0%sy, 0.0%ni, 94.3%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 1035124k total, 841168k used, 193956k free, 71316k buffers
Swap: 2096472k total, 84960k used, 2011512k free, 217816k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23889 root 16 0 2752 1032 676 R 3.0 0.1 0:00.04 top
1 root 18 0 2168 624 596 S 0.0 0.1 0:02.48 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.25 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.40 events/0
6 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
7 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
10 root 10 -5 0 0 0 S 0.0 0.0 0:10.34 kblockd/0
11 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
173 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0
176 root 10 -5 0 0 0 S 0.0 0.0 0:00.02 khubd
178 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 kseriod
240 root 19 0 0 0 0 S 0.0 0.0 0:00.01 khungtaskd
242 root 16 0 0 0 0 S 0.0 0.0 3:22.49 pdflush
243 root 10 -5 0 0 0 S 0.0 0.0 0:07.42 kswapd0
244 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
325 root 16 0 4916 1488 1236 S 0.0 0.1 0:00.09 bash
462 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
492 root 10 -5 0 0 0 S 0.0 0.0 0:00.27 mpt_poll_0
493 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 mpt/0
494 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
497 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0
The CPU(s) line displays current CPU usage. The id field indicates how much of the CPU is idle. If this number is very low, it may mean you don't have enough CPU resources for PGP Universal Server. Upgrading the CPU or allocating additional CPU resources (for virtual machines) will help with this.
The Mem line displays current RAM usage. If the free field is low, it may indicate that you need to add or allocate additional memory to the server. Because PGP Universal Server uses PAE kernel extensions, it can support up to 64GB of RAM despite using a 32-bit version of Linux.
The Swap line displays current swap file usage in much the same way that RAM is displayed. If the Mem free is low and the Swap used is high, you will see a significant decrease in performance and should add or allocation additional RAM as soon as possible. Excessive swapping can also lead to system lockups as the system becomes non-responsive.
Beneath the swap line is a list of currently running processes. By deault, this will be sorted by CPU usage and will refresh about every 4 seconds. This can help identify which processes are using the most CPU time. By using the greater than (>) and less than (<) keys on your keyboard, you can change the order in which processes are sorted. From the default view, the greater than key will change the sort order to memory usage.
Once you are done reviewing the information in top, press q to quit the program.
sar
The sar utility will display history system usage statistics including CPU usage, RAM usage, and disk activity. This can help pinpoint specific bottlenecks in performance. PGP Universal Server retains data over a seven day period to help isolate potential hardware utilization issues.
To display CPU usage for the current day, run sar without any arguments. You will get output similar to the following:
Linux 2.6.18-194.26.1.el5PAE (keys.lab.jharris.local) 11/03/2011
12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 2.08 0.00 4.43 0.30 0.00 93.18
12:20:01 AM all 1.45 0.00 3.80 0.03 0.00 94.72
12:30:01 AM all 1.42 0.00 3.84 0.02 0.00 94.72
12:40:01 AM all 1.45 0.00 3.99 0.03 0.00 94.54
12:50:02 AM all 1.52 0.00 3.67 0.04 0.00 94.78
01:00:01 AM all 1.53 0.00 3.84 0.02 0.00 94.62
01:10:01 AM all 1.56 0.00 3.91 0.02 0.00 94.51
01:20:01 AM all 1.57 0.00 3.90 0.04 0.00 94.49
01:30:02 AM all 1.40 0.00 3.76 0.02 0.00 94.82
01:40:01 AM all 1.88 0.00 4.44 0.04 0.00 93.64
01:50:01 AM all 1.50 0.00 3.89 0.03 0.00 94.58
02:00:01 AM all 1.46 0.00 3.71 0.03 0.00 94.80
02:10:01 AM all 1.57 0.00 3.77 0.02 0.00 94.64
02:20:01 AM all 1.55 0.00 3.84 0.02 0.00 94.59
02:30:01 AM all 1.49 0.00 3.79 0.02 0.00 94.70
02:40:01 AM all 1.48 0.00 3.69 0.02 0.00 94.81
02:50:01 AM all 1.51 0.00 3.89 0.02 0.00 94.57
03:00:01 AM all 1.45 0.00 3.84 0.02 0.00 94.68
03:10:01 AM all 6.63 0.00 6.72 0.46 0.00 86.20
03:20:01 AM all 1.52 0.00 5.00 0.02 0.00 93.47
03:30:02 AM all 1.45 0.00 4.88 0.03 0.00 93.64
03:40:01 AM all 1.35 0.00 4.94 0.02 0.00 93.68
03:50:01 AM all 1.47 0.00 5.05 0.02 0.00 93.46
04:00:01 AM all 1.43 0.00 4.90 0.02 0.00 93.65
04:10:01 AM all 1.64 0.00 5.30 1.28 0.00 91.78
04:20:01 AM all 1.46 0.00 4.72 0.02 0.00 93.80
04:30:01 AM all 1.41 0.00 4.82 0.03 0.00 93.74
04:40:01 AM all 1.51 0.00 4.90 0.03 0.00 93.56
04:50:02 AM all 1.36 0.00 4.79 0.02 0.00 93.83
05:00:01 AM all 1.45 0.00 4.84 0.02 0.00 93.69
05:10:01 AM all 1.48 0.00 4.88 0.03 0.00 93.62
05:20:01 AM all 1.41 0.00 4.70 0.02 0.00 93.86
05:30:01 AM all 1.53 0.00 4.74 0.02 0.00 93.72
05:40:01 AM all 1.48 0.00 4.97 0.09 0.00 93.46
05:50:01 AM all 1.51 0.00 4.90 0.03 0.00 93.56
06:00:01 AM all 1.42 0.00 4.67 0.03 0.00 93.88
06:10:01 AM all 1.47 0.00 4.92 0.04 0.00 93.57
06:20:01 AM all 1.55 0.00 4.68 0.03 0.00 93.74
06:30:02 AM all 1.45 0.00 4.69 0.03 0.00 93.83
06:40:01 AM all 1.71 0.00 5.07 0.02 0.00 93.20
06:50:01 AM all 1.74 0.00 5.06 0.02 0.00 93.18
07:00:02 AM all 1.46 0.00 4.56 0.03 0.00 93.95
07:10:02 AM all 1.73 0.00 4.71 0.05 0.00 93.50
Average: all 1.64 0.00 4.50 0.07 0.00 93.79
This displays the CPU usage every 10 minutes for the current day. The idle column shows how much CPU is unused. A low number could indicate that the CPU is not powerful enough to process all of the requests coming to the server. The user and system columns show how much CPU is being used by programs and the OS, respectively.
The iowait column indicates how much CPU time was spent waiting for requested data to be fetched from the hard drive. A high number can indicate that the hard drive is too slow. This can be resolved either with a faster hard drive or by building a RAID array to increase performance. RAID 0, 5, 10, and 50 all provide increased disk performance. It may also indicate a problem with drive health such as bad sectors or a failing drive. If this is the case, the dmesg log (/var/log/dmesg) may show error messages related to the drive. You may also see messages about running a file system check when the server is booting up.
You can use sar to help identify these kinds of disk bottlenecks. Run sar -W to get a list of disk swapping activity. Using sar -b will show how many transactions per second are hitting the hard drive. These can both help isolate swapping and drive performance issues, including if issues happen during a certain peak time.
You can also use sar to pinpoint memory or swap utilization issues. Run sar -r and you will get output similar to the following:
Linux 2.6.18-194.26.1.el5PAE (keys.lab.jharris.local) 11/03/2011
12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
12:10:01 AM 143368 891756 86.15 102852 231664 2014236 82236 3.92 18760
12:20:01 AM 143740 891384 86.11 103128 231792 2014236 82236 3.92 18760
12:30:01 AM 142624 892500 86.22 103424 231840 2014236 82236 3.92 18760
12:40:01 AM 142624 892500 86.22 103756 231924 2014236 82236 3.92 18760
12:50:02 AM 142376 892748 86.25 103960 232056 2014236 82236 3.92 18760
01:00:01 AM 138904 896220 86.58 104232 232104 2014236 82236 3.92 18760
01:10:01 AM 141384 893740 86.34 104512 232252 2014236 82236 3.92 18760
01:20:01 AM 140640 894484 86.41 104784 232412 2014236 82236 3.92 18760
01:30:02 AM 135060 900064 86.95 105028 232464 2014236 82236 3.92 18760
01:40:01 AM 139100 896024 86.56 105304 232948 2014236 82236 3.92 18920
01:50:01 AM 138356 896768 86.63 105512 233084 2014236 82236 3.92 18920
02:00:01 AM 133148 901976 87.14 105756 233256 2014236 82236 3.92 19020
02:10:01 AM 138852 896272 86.59 106000 233368 2014236 82236 3.92 19020
02:20:01 AM 138604 896520 86.61 106184 233484 2014236 82236 3.92 19020
02:30:01 AM 136496 898628 86.81 106420 233516 2014236 82236 3.92 19020
02:40:01 AM 137984 897140 86.67 106640 233656 2014236 82236 3.92 19020
02:50:01 AM 137488 897636 86.72 106884 233764 2014236 82236 3.92 19020
03:00:01 AM 133644 901480 87.09 107068 233812 2014236 82236 3.92 19020
03:10:01 AM 231464 803660 77.64 54184 196256 2011508 84964 4.05 19172
03:20:01 AM 231232 803892 77.66 54756 196584 2011508 84964 4.05 19172
03:30:02 AM 229504 805620 77.83 55324 196608 2011508 84964 4.05 19172
03:40:01 AM 229820 805304 77.80 55860 196736 2011508 84964 4.05 19172
03:50:01 AM 228948 806176 77.88 56440 196832 2011508 84964 4.05 19172
04:00:01 AM 226884 808240 78.08 56968 196880 2011508 84964 4.05 19172
04:10:01 AM 203340 831784 80.36 64196 214652 2011508 84964 4.05 19172
04:20:01 AM 201976 833148 80.49 64728 214760 2011508 84964 4.05 19172
04:30:01 AM 202224 832900 80.46 65288 214784 2011508 84964 4.05 19172
04:40:01 AM 200860 834264 80.60 65756 214904 2011508 84964 4.05 19172
04:50:02 AM 201108 834016 80.57 66264 215000 2011508 84964 4.05 19172
05:00:01 AM 197016 838108 80.97 66848 215052 2011508 84964 4.05 19172
05:10:01 AM 198256 836868 80.85 67360 215172 2011508 84964 4.05 19172
05:20:01 AM 199496 835628 80.73 67868 215272 2011508 84964 4.05 19172
05:30:01 AM 199000 836124 80.78 68372 215304 2011508 84964 4.05 19172
05:40:01 AM 196516 838608 81.02 68968 216768 2011508 84964 4.05 19172
05:50:01 AM 195772 839352 81.09 69428 216932 2011512 84960 4.05 19200
06:00:01 AM 190812 844312 81.57 69960 216960 2011512 84960 4.05 19200
06:10:01 AM 195100 840024 81.15 70488 217348 2011512 84960 4.05 19200
06:20:01 AM 194604 840520 81.20 70976 217444 2011512 84960 4.05 19200
06:30:02 AM 190772 844352 81.57 71520 217832 2011512 84960 4.05 19200
06:40:01 AM 192384 842740 81.41 72068 217964 2011512 84960 4.05 19200
06:50:01 AM 191020 844104 81.55 72636 218052 2011512 84960 4.05 19200
07:00:02 AM 188644 846480 81.78 73084 218140 2011512 84960 4.05 19200
07:10:02 AM 185420 849704 82.09 73632 220064 2011512 84960 4.05 19200
Average: 176897 858227 82.91 82196 220505 2012651 83821 4.00 19055
The columns are, in order, RAM free, RAM used, percentage of RAM used, RAM used for buffers, RAM used for caching, swap free, swap used, percentage of swap used, and swap cached. A high %memused and/or %swpused indicate that you need more RAM to improve performance.
df
The df utility is used to show how much drive space is available and how much of it is being used. Disks with low free space may perform worse as file fragmentation will often be higher. Here is a sample of output using df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 37G 3.5G 32G 11% /
/dev/sda1 99M 17M 77M 18% /boot
tmpfs 506M 8.0K 506M 1% /dev/shm
In this example, the main partition on the hard drive, /dev/sda2, is 37GB with 11% in use. The -h argument shows the output in a friendly human-readable format. If you are running into disk space issues, you should consider upgrading to a larger hard drive or expanding your RAID array, if you have one.