Health: Critical - Investigation and Fix

Products

• SG-S400
• ProxySG Software - SGOS

Issue/Introduction

Health check on the Edge SWG (ProxySG) shows : Critical

Stat: DIMM A1 temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:42:55 UTC
Current Value                 : No Input
High Critical Threshold       : 76.0
High Warning Threshold        : 73.0
Low Warning Threshold         : 3.0
Low Critical Threshold        : 1.0
Notification Method           : log

Stat: DIMM A2 temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:42:55 UTC
Current Value                 : No Input
High Critical Threshold       : 76.0
High Warning Threshold        : 73.0
Low Warning Threshold         : 3.0
Low Critical Threshold        : 1.0
Notification Method           : log

Stat: Front panel temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:05 UTC
Current Value                 : No Input
High Critical Threshold       : 55.0
High Warning Threshold        : 53.0
Low Warning Threshold         : -2.0
Low Critical Threshold        : -4.0
Notification Method           : log

Stat: Midplane center temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:15 UTC
Current Value                 : No Input
High Critical Threshold       : 57.0
High Warning Threshold        : 54.0
Low Warning Threshold         : -2.0
Low Critical Threshold        : -4.0
Notification Method           : log

Stat: Midplane left temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:15 UTC
Current Value                 : No Input
High Critical Threshold       : 57.0
High Warning Threshold        : 54.0
Low Warning Threshold         : -2.0
Low Critical Threshold        : -4.0
Notification Method           : log

Stat: Midplane right temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:25 UTC
Current Value                 : No Input
High Critical Threshold       : 60.0
High Warning Threshold        : 57.0
Low Warning Threshold         : -2.0
Low Critical Threshold        : -4.0
Notification Method           : log

Stat: PCH temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:25 UTC
Current Value                 : No Input
High Critical Threshold       : 101.0
High Warning Threshold        : 98.0
Notification Method           : log

Stat: Power supply inlet temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:25 UTC
Current Value                 : No Input
High Critical Threshold       : 59.0
High Warning Threshold        : 56.0
Low Warning Threshold         : -2.0
Low Critical Threshold        : -4.0
Notification Method           : log

Stat: SAS controller temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:25 UTC
Current Value                 : No Input
High Critical Threshold       : 69.0
High Warning Threshold        : 66.0
Low Warning Threshold         : -2.0
Low Critical Threshold        : -4.0
Notification Method           : log

Stat: System center temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:35 UTC
Current Value                 : No Input
High Critical Threshold       : 70.0
High Warning Threshold        : 67.0
Low Warning Threshold         : -2.0
Low Critical Threshold        : -4.0
Notification Method           : log

Stat: System left temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:35 UTC
Current Value                 : No Input
High Critical Threshold       : 67.0
High Warning Threshold        : 64.0
Low Warning Threshold         : -2.0
Low Critical Threshold        : -4.0
Notification Method           : log

Stat: System right temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:35 UTC
Current Value                 : No Input
High Critical Threshold       : 74.0
High Warning Threshold        : 71.0
Low Warning Threshold         : -2.0
Low Critical Threshold        : -4.0
Notification Method           : log

Stat: CPU temperature
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:42:55 UTC
Current Value                 : No Input
High Critical Threshold       : 96.0
High Warning Threshold        : 94.0
Low Warning Threshold         : 4.0
Low Critical Threshold        : 2.0
Notification Method           : log

Stat: System fan 1 front speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:45 UTC
Current Value                 : No Input
Low Warning Threshold         : 4500.0
Low Critical Threshold        : 4000.0
Notification Method           : log

Stat: System fan 1 rear speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:55 UTC
Current Value                 : No Input
Low Warning Threshold         : 3500.0
Low Critical Threshold        : 3000.0
Notification Method           : log

Stat: System fan 2 front speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:55 UTC
Current Value                 : No Input
Low Warning Threshold         : 4500.0
Low Critical Threshold        : 4000.0
Notification Method           : log

Stat: System fan 2 rear speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:43:55 UTC
Current Value                 : No Input
Low Warning Threshold         : 3500.0
Low Critical Threshold        : 3000.0
Notification Method           : log

Stat: System fan 3 front speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:05 UTC
Current Value                 : No Input
Low Warning Threshold         : 4500.0
Low Critical Threshold        : 4000.0
Notification Method           : log

Stat: System fan 3 rear speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:05 UTC
Current Value                 : No Input
Low Warning Threshold         : 3500.0
Low Critical Threshold        : 3000.0
Notification Method           : log

Stat: System fan 4 front speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:05 UTC
Current Value                 : No Input
Low Warning Threshold         : 4500.0
Low Critical Threshold        : 4000.0
Notification Method           : log

Stat: System fan 4 rear speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:05 UTC
Current Value                 : No Input
Low Warning Threshold         : 3500.0
Low Critical Threshold        : 3000.0
Notification Method           : log

Stat: System fan 5 front speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:05 UTC
Current Value                 : No Input
Low Warning Threshold         : 4500.0
Low Critical Threshold        : 4000.0
Notification Method           : log

Stat: System fan 5 rear speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:15 UTC
Current Value                 : No Input
Low Warning Threshold         : 3500.0
Low Critical Threshold        : 3000.0
Notification Method           : log

Stat: System fan 6 front speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:15 UTC
Current Value                 : No Input
Low Warning Threshold         : 4500.0
Low Critical Threshold        : 4000.0
Notification Method           : log

Stat: System fan 6 rear speed
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:15 UTC
Current Value                 : No Input
Low Warning Threshold         : 3500.0
Low Critical Threshold        : 3000.0
Notification Method           : log

Stat: +1.1V standby voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:15 UTC
Current Value                 : No Input
High Critical Threshold       : 1.1956
High Warning Threshold        : 1.1662
Low Warning Threshold         : 1.0290
Low Critical Threshold        : 1.0094
Notification Method           : log

Stat: +12V main bus 1 voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:15 UTC
Current Value                 : No Input
High Critical Threshold       : 13.065
High Warning Threshold        : 12.675
Low Warning Threshold         : 11.245
Low Critical Threshold        : 10.920
Notification Method           : log

Stat: +12V main bus 2 voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:15 UTC
Current Value                 : No Input
High Critical Threshold       : 13.065
High Warning Threshold        : 12.675
Low Warning Threshold         : 11.245
Low Critical Threshold        : 10.920
Notification Method           : log

Stat: +3.3V main bus voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:25 UTC
Current Value                 : No Input
High Critical Threshold       : 3.580
High Warning Threshold        : 3.508
Low Warning Threshold         : 3.076
Low Critical Threshold        : 3.004
Notification Method           : log

Stat: +3.3V standby voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:25 UTC
Current Value                 : No Input
High Critical Threshold       : 3.583
High Warning Threshold        : 3.511
Low Warning Threshold         : 3.079
Low Critical Threshold        : 3.007
Notification Method           : log

Stat: +5V main bus voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:25 UTC
Current Value                 : No Input
High Critical Threshold       : 5.4372
High Warning Threshold        : 5.2992
Low Warning Threshold         : 4.6920
Low Critical Threshold        : 4.5540
Notification Method           : log

Stat: +5V standby voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:25 UTC
Current Value                 : No Input
High Critical Threshold       : 5.4372
High Warning Threshold        : 5.2992
Low Warning Threshold         : 4.6920
Low Critical Threshold        : 4.5540
Notification Method           : log

Stat: +3V battery voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:25 UTC
Current Value                 : No Input
High Critical Threshold       : 3.624
High Warning Threshold        : 3.504
Low Warning Threshold         : 2.544
Low Critical Threshold        : 2.448
Notification Method           : log

Stat: BMC memory voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:35 UTC
Current Value                 : No Input
High Critical Threshold       : 1.672
High Warning Threshold        : 1.624
Low Warning Threshold         : 1.480
Low Critical Threshold        : 1.472
Notification Method           : log

Stat: BMC PLL voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:35 UTC
Current Value                 : No Input
High Critical Threshold       : 1.360
High Warning Threshold        : 1.336
Low Warning Threshold         : 1.184
Low Critical Threshold        : 1.144
Notification Method           : log

Stat: CPU core voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:35 UTC
Current Value                 : No Input
High Critical Threshold       : 1.3818
High Warning Threshold        : 1.3524
Low Warning Threshold         : 0.5880
Low Critical Threshold        : 0.5586
Notification Method           : log

Stat: CPU PLL voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:45 UTC
Current Value                 : No Input
High Critical Threshold       : 1.9600
High Warning Threshold        : 1.9012
Low Warning Threshold         : 1.5876
Low Critical Threshold        : 1.5484
Notification Method           : log

Stat: CPU system agent voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:45 UTC
Current Value                 : No Input
High Critical Threshold       : 1.2250
High Warning Threshold        : 1.2152
Low Warning Threshold         : 0.5782
Low Critical Threshold        : 0.5684
Notification Method           : log

Stat: CPU termination voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:45 UTC
Current Value                 : No Input
High Critical Threshold       : 1.136
High Warning Threshold        : 1.112
Low Warning Threshold         : 0.928
Low Critical Threshold        : 0.912
Notification Method           : log

Stat: Memory I/O voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:45 UTC
Current Value                 : No Input
High Critical Threshold       : 1.632
High Warning Threshold        : 1.584
Low Warning Threshold         : 1.408
Low Critical Threshold        : 1.360
Notification Method           : log

Stat: Memory termination voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:45 UTC
Current Value                 : No Input
High Critical Threshold       : 0.816
High Warning Threshold        : 0.792
Low Warning Threshold         : 0.704
Low Critical Threshold        : 0.680
Notification Method           : log

Stat: PCH core voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:45 UTC
Current Value                 : No Input
High Critical Threshold       : 1.192
High Warning Threshold        : 1.168
Low Warning Threshold         : 1.024
Low Critical Threshold        : 1.000
Notification Method           : log

Stat: PCH SAS voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:55 UTC
Current Value                 : No Input
High Critical Threshold       : 1.632
High Warning Threshold        : 1.584
Low Warning Threshold         : 1.408
Low Critical Threshold        : 1.360
Notification Method           : log

Stat: SAS core voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:44:55 UTC
Current Value                 : No Input
High Critical Threshold       : 1.120
High Warning Threshold        : 1.104
Low Warning Threshold         : 0.968
Low Critical Threshold        : 0.944
Notification Method           : log

Stat: SAS I/O voltage
Current State                 : CRITICAL
Last Transition               : Tue, 28 Sep 2021 00:42:35 UTC
Current Value                 : No Input
High Critical Threshold       : 1.9600
High Warning Threshold        : 1.9012
Low Warning Threshold         : 1.6856
Low Critical Threshold        : 1.6366
Notification Method           : log

Stat: Power supply 1 status
Current State                 : WARNING
Last Transition               : Tue, 28 Sep 2021 00:42:45 UTC
Current Value                 : Undetected
Notification Method           : log

Stat: Power supply 2 status
Current State                 : WARNING
Last Transition               : Tue, 28 Sep 2021 00:42:45 UTC
Current Value                 : Undetected
Notification Method           : log

 

Cause

Having investigated the logs, we confirmed the following, from the diagnostics tool.

It was confirmed that a restart of the appliance resolved the CRITICAL errors.

Having investigated further, we confirmed that the issue relates to the BMC chip going out-of-sync, and by restarting the appliance, the BMC chip was reset and that cleared the errors.

 

Resolution

Further checks into the sysinfo log showed that the BMC firmware on the reported S400-20 appliance is on version 3.16 and a bit older. See excerpt below, from the sysinfo file.

And the latest version of BMC firmware, for the same appliance model, is 3.21. See the Tech. Article with URL below, for reference.

https://knowledge.broadcom.com/external/article/170602/finding-which-bmc-firmware-version-is-on.html

Resolution: To ensure this CRITICAL issue does not recur, we recommend an upgrade of the BMC firmware on the appliance, to the latest version.

Note: The BMC firmware is embedded in hardware diagnostics version 3.1.2.1 or later, i.e. there is no separate BMC firmware to download

To upgrade the BMC firmware to the latest version, an upgrade of the HW diagnostics will be required. For this appliance model, the HW diagnostics should be upgraded to version 3.1.3.1. For the download links for the HW diagnostics as well as the install instructions, please refer to the Tech. Article with URL below.

https://knowledge.broadcom.com/external/article/169399/how-to-upgrade-hardware-diagnostics-and.html