Cisco UCS B-Series blade is stuck booting at "activating: system-storage."
search cancel

Cisco UCS B-Series blade is stuck booting at "activating: system-storage."

book

Article ID: 441933

calendar_today

Updated On:

Products

VMware vSphere ESX 8.x

Issue/Introduction

When attempting to manage or boot an ESXi host running on a Cisco UCS B-Series blade server, the host may enter a "Not Responding" state in vCenter Server or experience severe local filesystem hangs.

During the boot phase, the system stalls indefinitely on the grey and yellow Direct Console User Interface (DCUI) screen at the following stage:

  • activating: system-storage

Press F12 to Review the ESXi-/var/log/vmkernel.log, the console logs to reveals the following repeating error strings:

2026-05-27T13:22:56.075Z cpu22:2098319)WARNING: Fil3: 1643: Failed to reserve volume f533 28 1 6520300b b6b99f52 2500b00f 1af022
2026-05-27T13:23:14.064Z cpu0:2097292)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "mpx.vmhba32:C0:T0:L0" stat
2026-05-27T13:23:14.064Z cpu0:2097292)ScsiDeviceIo: 4670: Cmd(0x45b9c26c2480) 0x2a, cmdId.initiator=0x4309fe323300 CmdSN 0x39 fr
2026-05-27T13:23:14.099Z cpu2:2098319)J6: 3238: Failed to initialize journal block for volume 'LOCKER-6520300b-b6b99f52-b00f-002
2026-05-27T13:23:18.094Z cpu2:2098319)Vol3: 4768: Failed to get object 28 type 1 uuid 6520300b-b6b99f52-b00f-0025b522f01a FD 0 g
2026-05-27T13:23:18.094Z cpu2:2098319)WARNING: Fil3: 1643: Failed to reserve volume f533 28 1 6520300b b6b99f52 2500b00f 1af022b
2026-05-27T13:23:36.060Z cpu0:2097292)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "mpx.vmhba32:C0:T0:L0" stat
2026-05-27T13:23:36.060Z cpu0:2097292)ScsiDeviceIo: 4670: Cmd(0x45b9c2762c80) 0x2a, cmdId.initiator=0x4309fe323300 CmdSN 0x39 fr
2026-05-27T13:23:36.096Z cpu2:2098319)J6: 3238: Failed to initialize journal block for volume 'LOCKER-6520300b-b6b99f52-b00f-002
2026-05-27T13:23:40.068Z cpu2:2098319)Vol3: 4768: Failed to get object 28 type 1 uuid 6520300b-b6b99f52-b00f-0025b522f01a FD 0 g
2026-05-27T13:23:40.068Z cpu2:2098319)WARNING: Fil3: 1643: Failed to reserve volume f533 28 1 6520300b b6b99f52 2500b00f 1af022b
2026-05-27T13:23:44.272Z cpu2:2098067)NMP: nmp_ResetDeviceLogThrottling:3854: last error status from device mpx.vmhba32:C0:T0:L0
2026-05-27T13:23:58.057Z cpu0:2097292)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x2a (0x45b9c2666080, 2098319) to dev "mpx.vmhba32
2026-05-27T13:23:58.057Z cpu2:2098319)NMP: nmp_ThrottleLogForDevice:3898: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x4309fe

Environment

vsphere 8.x

Cause

This issue occurs when the underlying storage path to the ESXi host's local boot or locker media is abruptly dropped or becomes desynchronized due to a transient management-plane failure on the physical hardware.

On Cisco UCS B-Series blades, if the Chassis Management Controller (CMC) or the onboard mezzanine/CIMC storage interface encounters a silent freeze or a communication failure with the Fabric Interconnects, the storage paths to local SAS/SATA/M.2 arrays drop mid-flight.

When ESXi attempts to write runtime metadata or place a SCSI reservation lock on the active volume, the missing storage target triggers a cascade failure:

  1. Journal Failure (J6): The VMFS filesystem driver cannot replay or allocate the transaction journal block needed to keep the system in a healthy read/write state.
  2. Metadata Loss (Vol3): Internal file descriptors (such as Object 28) lose their in-memory backing pointers, causing immediate system or storage-subsystem panics.

When the host is rebooted while in this state, the boot process hangs because the kernel enters an infinite I/O retry loop while trying to mount and resolve the corrupted filesystem journal pointers.

Resolution

To resolve this issue, you must reset the physical hardware slot to restore the communication paths to the storage controller, break the infinite boot loop, and then verify the integrity of the underlying VMFS filesystem.

Step 1: Recover Hardware Connectivity via Cisco UCS Manager CLI

Before attempting any software or driver adjustments inside ESXi, perform a non-disruptive slot-level management reset via the UCS Manager CLI to wake up the unresponsive chassis slot.

  1. Establish an SSH connection to the active Cisco UCS Fabric Interconnect.

  2. Enter the following command sequence to target the specific chassis and server slot exhibiting the failure (e.g., Chassis 4, Slot 7):

    scope chassis 4
scope server 7
reset slot

  3. Acknowledge the prompt to execute the slot reset. Monitor the progress of the Fabric Subsystem Manager loop via the following command:

    show fsm status
  4. Allow the server discovery loop to hit 100% and return to an "OK" status.

Step 2: Force a Hardware Cold Boot (If Still Hung)

If the host remains frozen on the yellow screen after the slot reset, a hard cold power cycle must be issued to completely drain and re-initialize the blade's stuck PCIe/storage bus state.

  1. In the UCS Manager CLI, forcefully stop power to the blade:

    scope chassis 4
scope server 7
power stop immediate

  2. Wait approximately 60 seconds for the volatile cache on the onboard controllers to clear.

  3. Power the server back on: 

    power start
 
Powered by Wolken Software