Target Disk Busy error, "I/O error"

Products

• VMware Tanzu Data Suite
• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

When attempting to recover Greenplum segments using gprecoverseg or gprecoverseg -F, the operation fails. Reviewing the system logs or attempting manual intervention reveals the following:

• Error Message: umount: /data/volX: target is busy when attempting to unmount the segment disk.
• I/O Errors: System dmesg or /var/log/syslog shows Input/output error or XFS (sdX): metadata I/O error for the specific partition.
• Disk State: The filesystem may be reported as 'Read-only' or 'Busy,' and standard commands like ls on the mount point hang or return I/O errors.

Environment

All Linux OS versions

Cause

This issue typically occurs when the Linux kernel loses communication with the local disk partition (e.g., /dev/sda8) while processes are still attempting to access it.

Standard umount commands fail because background Greenplum processes or kernel threads are stuck in an 'Uninterruptible Sleep' (D state) while waiting for I/O that will never complete. Because these processes cannot be killed, the 'target is busy' error persists even after stopping Greenplum services.

Resolution

Follow these steps to clear the hardware lock and recover the segments:

1. Identify the Faulty Device: Check which partition is failing by running:

   bash

    

   dmesg | grep -i "I/O error"

   Confirm if the mount point (e.g., /data/vol4) matches the segments that are down.

2. Attempt Service Termination: Try to stop the cluster and any remaining segment processes:

   bash

    

   gpstop -a

3. Perform a System Reboot: If the umount: target is busy error persists and lsof or fuser cannot clear the processes, a full system reboot of the affected segment host is required. This resets the hardware interface and clears stuck kernel processes.

4. Verify Mount and Hardware Health: After the reboot, ensure the disk has mounted correctly and is writable:

   bash

    

   mount | grep /data/volXtouch /data/volX/test_file && rm /data/volX/test_file

5. Recover Greenplum Segments: Once the disk is accessible, bring the cluster online and run the recovery:

   bash

    

   gpstart -agprecoverseg -a

   Note: If data was corrupted during the I/O failure, a full recovery (gprecoverseg -F) may be required.