Hard Drive failure on a Reporter appliance. Error: "WARNING - 23/24 RAID members are working"

Products

• Reporter-S500

Issue/Introduction

On your Reporter appliance, all of the hard drive (HDD) LEDs are showing green, but you received a message similar to "WARNING - 23/24 RAID members are working"   

Resolution

To resolve this issue, it requires the following steps:

1. 1. View the health monitoring status and confirm the warning exists:
      • reporter# configure t
      • reporter(config)# health-monitoring
      • reporter(config-health-monitoring)#view current
   2.  Display the state of the RAID array and hard disk drives that are part of the RAID to find out which drive is not active.
      • reporter# show raid array
      • reporter# show raid members
   3. Upload a diagnostic file to a support case and check the output from a file path: /sos_commands/md/mdadm_-D_.dev.md to locate a faulty HDD.
      # diagnostics service-info send-sr 123456789  

For example, when we execute the commands above, we can see that disk 20 has failed and needs to be replaced:

mdadm: /dev/md does not appear to be an md device
/dev/md127:
           Version : 1.2
     Creation Time : 
        Raid Level : raid10
        Array Size : 
     Used Dev Size : 
      Raid Devices : 24
     Total Devices : 23
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : 
             State : active, degraded 
    Active Devices : 23
   Working Devices : 23
    Failed Devices : 0
     Spare Devices : 0

            Layout : near=2
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : 
              UUID : 
            Events : 

    Number   Major   Minor   RaidDevice State
       0       8       49        0      active sync set-A   /dev/sdd1
       1       8      193        1      active sync set-B   /dev/sdm1
       2       8      209        2      active sync set-A   /dev/sdn1
       3       8      225        3      active sync set-B   /dev/sdo1
       4       8      241        4      active sync set-A   /dev/sdp1
       5      65        1        5      active sync set-B   /dev/sdq1
       6      65       17        6      active sync set-A   /dev/sdr1
       7      65       33        7      active sync set-B   /dev/sds1
       8      65       49        8      active sync set-A   /dev/sdt1
       9      65       65        9      active sync set-B   /dev/sdu1
      10      65       81       10      active sync set-A   /dev/sdv1
      11       8       65       11      active sync set-B   /dev/sde1
      12      65       97       12      active sync set-A   /dev/sdw1
      13      65      113       13      active sync set-B   /dev/sdx1
      14      65      129       14      active sync set-A   /dev/sdy1
      15      65      145       15      active sync set-B   /dev/sdz1
      16      65      161       16      active sync set-A   /dev/sdaa1
      17       8       81       17      active sync set-B   /dev/sdf1
      18       8       97       18      active sync set-A   /dev/sdg1
      19       8      113       19      active sync set-B   /dev/sdh1
      -        0        0       20      removed
      21       8      129       21      active sync set-A   /dev/sdi1
      22       8      145       22      active sync set-B   /dev/sdj1
      23       8      177       23      active sync set-B   /dev/sdl1

Additional Information

WARNING! Functioning hard drives in the RP-S500 are not hot-swappable. Failed hard drives are not hot-swappable.

The system will automatically start the RAID rebuild process after replacing the disk and powering up the appliance.  Once the hard drive is replaced, It takes time to recover the RAID and it's normal to see the message "RAID state: active, degraded, recovering: 55% complete" during the RAID rebuild process.