Data to gather if PacketShaper is hung, blocking all network traffic, and the PacketShaper UI is inaccessible

Products

• PacketShaper S-Series

Issue/Introduction

PacketShaper is not responding to CLI/WUI and is blocking all Network traffic going through it.

Environment

PacketShaper is connected Inline between LAN and WAN, usually towards the edge of the WAN. All WAN traffic from LAN traverses through PacketShaper.

Cause

PacketShaper is in a hung state and is unable to process any traffic going through it. It may not be able to respond to IP communication to the PS.

Resolution

When the PacketShaper is in hung or unresponsive state, there may not be much you can do to recover. However, it is very important to find the root cause of the hang so that the issue can be fixed and we do not run into the same issue again. In order to do proper RCA, we need to collect as much information as possible during the hung state.

Before we conclude it is a true hang find out the following:

1. Is traffic completely blocked through the PS?
2. Is all traffic blocked through all NICs or only though one of the NICs?
3. What is the status of the LINK lights, are they ON, OFF, Blinking, etc. Sometimes, traffic will be blocked if one of the ports goes down and that may not necessarily be a hang.
4. Is PS accessible via SSH and WUI?
5. Any other observation during the hang?

Once we gather the above information, please use one of the methods below to generate the core file and send to Support:

A. If you have access to the PacketShaper via Serial Console:

1. Login to PS via 'console' into Recovery Mode. The username and password to login to recovery mode is 'recovery/recovery'.
2. If console access is not working, please try 'Ctrl+\' and also 'Ctrl+C then 'Ctrl+Z' to refresh/release any disconnected console sessions and try again.
3. After you login to console into Recovery mode, you will see the Menu below:

PACKETSHAPER login: recovery
Password:
--------------------------------------------------------------------------
Please select one of the following recovery actions
0. Exit
1. Restore to default configuration
2. Restore to default touch user password
3. Regenerate SSL certificate
4. Reformat system hard drive
5. Wipe off system hard drive
6. Soft restart (generate core dump)
7. Reboot
8. Shutdown
[0 - 8]

4. Select Option '6. Soft restart (generate core dump)'
5. The PS will restart and save the core files under 9.256/log folder.
6. Please make sure that you allow enough time for the core write to complete, do not restart again.
7. After unit reboots, please collect the contents of 9.256/ and 9.258/ from the WUI (Info, FileBrowser).
8. Open a case with Symantec Support and submit the above details.

B. If you do not have access to the PS via Serial Console:

Use the Keypad to Regain Console Access

Starting in PS v11.10.2.3, the keypad for the PacketShaper LCD can be used to regain serial console access after
PacketShaper enters a hung state. You can use this method as an alternative to generating the core dump using the
serial console when the appliance is hung. Because the serial console and front panel use separate, parallel channels
to interact with the system, the front panel can be used to regain control of the system if the serial console hangs.

The front panel offers the following modes:
1: Rls Console—Releases all login sessions to PacketShaper including the serial console and any
SSH sessions in remote login utilities. Mode 1 also generates a core of the serial console control
program if it is running. This mode does not impact the traffic process.

2: Collect Cores—In addition to the actions performed in mode 1, mode 2 generates a core of the
PacketShaper process. Note that traffic will be blocked during the core collection in mode 2.
All cores are generated in the 9.256/log folder. You can send the files to Symantec Support for help in diagnosing the
issue that caused the system to hang.

3: Kernel Dump - In addition to the actions performed in modes 1 and 2, mode 3 will dump kernel
memory. Traffic will be blocked during this process.
The upload is similar to the standard PS dump, but from 9.258.kernel.

To use the keypad, press the left or right arrow keys to cycle between the modes. When the desired mode is
displayed in the LCD, press the Enter key on the keypad; PacketShaper will then begin executing the actions
associated with the displayed mode.

Lastly, if the console does not respond as expected, press up+down+enter on the front panel at the same
time to perform an NMI dump. The NMI dump will also be written to the 9.258/kernel. Please send the contents of 9.258/kernel to Symantec Support for analysis.