Security Analytics on Dell r640xl server not booting after upgrade attempt

Products

• Security Analytics

Issue/Introduction

It is Critical that the BIOS be updated in all Dell r640 servers supplied by Broadcom using the Security Analytics application.  If you do not update the firmware, in some cases the system will not come back up during an upgrade and will require a field service technician to come out to the site and re-seat a DIMM. To prevent this, the BIOS will need to be updated.  This will require downtime with a reboot.

If there has been a previous failure on a DIMM, the system may not come back up after the reboot during a Security Analytics upgrade.

There is also a CVE against the iDRAC Lifecycle Controller version 5.X, DSA-2024-223.

Environment

Security Analytics systems running on Dell r640xl servers with versions previous to 2.18.1.  You can check all firmware versions for the r640 by logging in as root and running racadm getversion. If it returns something less than 2.18.1, you should update the BIOS before the next Security Analytics update. The current version is 2.21.1.

Cause

The Security Analytics server based on a Dell r640xl may be in a continuous boot loop due to a BIOS firmware bug in the hardware. If you do not update the firmware, in some cases the system will not come back up and will require a field service technician to come out to the site and re-seat a DIMM.

Resolution

You will need to download the BIOS update from Dell and upload it to the r640xl server.

 

The first step is to find the Asset Tag/Serial number.  It is in found in Security Analytics in the top right corner in the information "About" icon.  Click on the icon and the serial number will be displayed. The serial number will be seven characters, alpha-numeric.

The serial number is also found on a pullout tag on the front of the server on the left side.  Look for a blue tab, facing up.  Another place to look is to run as root, the command dmidecode -s chassis-serial-number.

To retrieve the BIOS, PERC H730, and iDRAC firmware files,

1. Go to https://www.dell.com/support/home/en-us/product-support/product/poweredge-r640/drivers
2. Select in the "Operating System" box, Red Hat Enterprise Linux 7.
3. Select Show all.
   
   

BIOS, PERC H730, and iDRAC Firmware Updates by Command Line (Recommended)

1. Select download to the right of "Dell Server PowerEdge BIOS R740/R740xd/R640/R940/7920R Version 2.21.2". This will download the "BIOS_72VRD_LN64_2.21.2.BIN" file.
2. Select download to the right of iDRAC 7.00.00.171 to download iDRAC-with-Lifecycle-Controller_Firmware_TNYR2_LN64_7.00.00.171_A00.BIN
3. Select download to the right of Dell PERC H730/H730P/H830/FD33xS/FD33xD Mini/Adapter RAID Controllers firmware version 25.5.9.0001 to download SAS-RAID_Firmware_700GG_LN_25.5.9.0001_A17.BIN
4. Copy these three files to your server, to the /home directory.  For example, /home/BIOS_YM8R4_LN64_2.18.1.BIN.
5. Login as root and shutdown Security Analytics with scotus stop.  
6. Run  bash /home/BIOS_72VRD_LN64_2.21.2.BIN
7. When this completes, it will reboot
8. For the PERC H730 firmware, a reboot will be required
9. Login as root
10. shutdown Security Analytics services with scotus stop.  
11. Run bash /home/SAS-RAID_Firmware_700GG_LN_25.5.9.0001_A17.BIN
12. The system will reboot
13. For the PERC iDRAC firmware a reboot will not be required,
14. Login as root
15. Run bash /home/iDRAC-with-Lifecycle-Controller_Firmware_TNYR2_LN64_7.00.00.171_A00.BIN
16. The version of the BIOS can be found as root by running racadm getversion

BIOS Firmware Update by iDRAC

An alternative is to download the iDRAC-with-Lifecycle-Controller_Firmware_TNYR2_LN64_7.00.00.171_A00.EXE file and login to the iDRAC on the server. 

1. Select download to the right of "Dell Server PowerEdge BIOS R740/R740xd/R640/R940/7920R Version 2.20.1". This will download the "BIOS_YM8R4_WN64_2.18.1.EXE" file.
2. Before uploading the BIOS to the iDRAC, shutdown Security Analytics as root from the CLI with scotus stop.  You will need to log in to the iDRAC on the server and follow the instructions from Dell below:

 

1. Log into the iDRAC9 web interface
2. Go to Maintenance > System Update. The Manual Update page is displayed.
3. On the Manual Update tab, select Local as the Location Type.
   
   
   iDRAC9 Update Screen
4. Click Choose File, select the firmware image file for the required component, and then click Upload.
5. After the upload is complete, the Update Details section displays each firmware file that is uploaded to iDRAC and its status. If the firmware image file is valid and was successfully uploaded, the Contents column displays a (+) icon next to the firmware image file name. Expand the name to view the Device Name, Current, and Available firmware version information.
6. Select the required firmware file, and do one of the following:
   • For firmware images that do not require a host system reboot, click Install. For example, iDRAC firmware file.
   • For firmware images that require a host system reboot, click Install and Reboot or Install Next Reboot.
   • To cancel the firmware update, click Cancel.
   When you click Install, Install and Reboot, or Install Next Reboot, the message Updating Job Queue is displayed.
7. To display the Job Queue page, click Job Queue. Use this page to view and manage your pending firmware updates. You can click OK to refresh the current page to view the status of the firmware update.

 

Additional Information

This is from the Dell BIOS upgrade instructions at PowerEdge: How to Update Firmware Remotely Using the Integrated Dell Remote Access Controller (iDRAC) Web Interface.

Link to download the latest firmware files for the Dell PowerEdge R640:  https://www.dell.com/support/home/en-us/product-support/product/poweredge-r640/drivers