Identify and replace a hard drive that has generated alarms for hard drive error and broken RAID on CPM

Article Id: 167783

Status: Published

Updated On: 22-03-2018 16:27

Legacy Id: TECH243719

Products:

XOS

Issue/Introduction:

How To: For an CPM, identify and replace a hard drive that has generated alarms for hard drive error and broken RAIDWhen running "show module status cp<#>", you will see the status of the hard drive(s) and RAID status. In the example below, RAID is broken and the Second Hard Drive has a critical error:

CBS# show module status cp2

Slot 14:

Board Type CP8600

[truncated for brevity]
  Hard Disk                    250(GB)
  Second Hard Disk             250(GB)
  Flash                        NA
  Hard Drive Error             None
  Second Hard Drive Error      Critical Error <<<<<<<<<<<<<<<<<
RAID Status                  Broken, RAID 1 <<<<<<<<<<<<<<<<<

Cause:

This procedure can be used to identify and replace a bad hard drive on the related CPM generating the alarms.

Resolution:

When running "show module status cp<#>", you will see the status of the hard drive(s) and RAID status. In the example below, RAID is broken and the Second Hard Drive has a critical error:

CBS# show module status cp2

Slot 14:

Board Type CP8600

.
.
.
  Hard Disk                    250(GB)
  Second Hard Disk             250(GB)
  Flash                        NA
  Hard Drive Error             None
  Second Hard Drive Error      Critical Error <<<<<<<<<<<<<<<<<
RAID Status                  Broken, RAID 1 <<<<<<<<<<<<<<<<<

1. First, check and identify if both drives are recognized. You should see two SCSI Host entries, representing 2 hard drives are present and accessible. (If only one line is shown, then module will need to be reset and start with step 1. again to ensure both drives are seen. Refer to KB# 4524 for more information on Broken RAID-1 recovery if there is no hard drive error generated as that is more suitable solution).

Enter the "unix" prompt" and look at the /proc/scsi/scsi file:

[[email protected] soporte_ebtel]# cat /proc/scsi/scsi

Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: Hitachi HTE72505 Rev: PC4O
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: Hitachi HTE72505 Rev: PC4O
Type: Direct-Access ANSI SCSI revision: 05

2. Check RAID status and identify the missing disk in the RAID set . Output below show issue with SATA2 sdb1[2](F):

[[email protected]]# cat /proc/mdstat

Personalities : [raid0] [raid1]
md1 : active raid1 sdb1[2](F) sda1[0]
104320 blocks [2/1] [U_] <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

md5 : active raid1 sdb5[2](F) sda5[0]
8008256 blocks [2/1] [U_] <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

md6 : active raid1 sdb6[2](F) sda6[0]
236002752 blocks [2/1] [U_] <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

3. Run the smartclt test on the "sdb" disk and record any errors.

[email protected]]# smartctl -a /dev/sdb

smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8
Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Short INQUIRY response, skip product id
A mandatory SMART command failed: exiting. To continue, add one or more
'-T permissive' options

5. Remove the affected CPM , insert a new blank disk in the same slot as the failed drive and re-insert the CPM.

6. When the module boots up, run the following command from the Linux shell:
[email protected]]# /crossbeam/bin/xos-raid-add
The RAID reconstruction might take several hours.

7. Next, when running "show module status cp<#>", you will see the status of the hard drive(s) and RAID status. In the example below, RAID status is fixed and the Second Hard Drive error has been cleared.

POD15# show module status cp2

Slot 14:

Board Type CP8600

[truncated for brevity]
  Hard Disk                    250(GB)
  Second Hard Disk             250(GB)
  Flash                        NA
  Hard Drive Error             None
  Second Hard Drive Error None           <<<<<<<<<<<<<<<<<
RAID Status Active, RAID 1 <<<<<<<<<<<<<<<<<

SCREEN SHOOT OF CPM