Dealing with processes in state "D" (uninterruptible sleep) (usually IO)

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

There are processes for which ps aux shows state "D" (uninterruptible sleep). An example of this behavior is shown below:

[gpadmin@sdapsdw05 ~]$ ps aux | grep 26528
gpadmin  26528  0.0  0.0 395116 211960 ?       Ds   Sep17   4:14 /usr/local/greenplum/greenplum-db-4.2.7.1SY4/bin/postgres -D /data/gpdb_p6/gpseg37 -p 50005 -b 39 -z 320 --silent-mode=true -i -M quiescent -C 37

State "D" (uninterruptible sleep) means that the process is in kernel space (in a system call), attempting to perform IO. These processes will not respond to signals (or SIGKILL) and cannot be debugged with gdb or pstack.

 

Environment

Cause

This is a known Red Hat issue: After upgrading the kernel to 2.6.32-279.14.1.el6, observing high load average and found xfsaild in D state.

 

Resolution

This issue is fixed in kernel version "kernel-2.6.32-431.el6". If DCA, upgrade to a DCA version with a newer kernel (see JIRA PFRM-256). If not DCA, upgrade the kernel.
 

There is no way to terminate or debug these processes. A server reboot is the only way to clear them up. It is recommended to collect Operating System details, logs, and information about the state of the storage subsystem.