6X0 SD-WAN Edges power off and require a reboot to come back
search cancel

6X0 SD-WAN Edges power off and require a reboot to come back

book

Article ID: 320676

calendar_today

Updated On:

Products

VMware VMware SD-WAN by VeloCloud

Issue/Introduction

Symptoms:

On a rare occasion, an Edge 6x0 (610, 610-LTE, 601N, 620, 620N, 640, 640N, 680, 680N) may power off with no noticeable trigger.

This is most often observed when an Edge 6x0 is connected to a power source that experiences or passes certain power fluctuation (brownout condition). In other words, the issue is directly related to the power environment, to which the Edge is connected.

A power cycle (unplugging and then re-plugging the power to the 6x0 Edge) is required to bring the device back to a working state.

 

Environment

VMware SD-WAN
VMware SD-WAN by VeloCloud

Cause

This issue is triggered whenever there are rapid constant power outages or "flaps". The cause of the issue is traced to a PIC microcontroller exclusive to the Edge 6x0 line, which uses a PIC firmware version of v20M or earlier.

This is tracked as Issue #89217.

Resolution

The issue is resolved by upgrading the Platform Firmware, which includes PIC version v20N, on the affected Edge 6x0. Previously, this Knowledge Base article instructed customers to upgrade to Platform Firmware 1.3.0 (R130-20220328-GA); however, this image is no longer recommended and is now marked as deprecated in the SASE Orchestrators.

Version 1.3.0 is replaced by Platform Firmware 1.3.1 (R131-20221216-GA).

Note: The Orchestrator must be using Release 5.0.0 or higher before the Edge's software and firmware versions can be upgraded.

Edge Platform Firmware Upgrade Process

  1. The Edge needs to be upgraded to the R5014-20230713-GA version, this means:
    1. The Operator Profile lists a Software Version of R5014-20230713-GA.
    2. The Operator Profile must NOT include either a Firmware Version or a Factory Version, only the listed Software Version.
    3. The Edge Operations Team created and added this Operator Profile to all hosted Orchestrators running Orchestrator Release 5.x or higher.
Note: Prior to the April 4th update, the required Edge software version was R5012-20230123-GA-103475. That Edge version was replaced by version R5012-20230327-GA-107522 on July 14th, and the prior version was marked as deprecated on the VMware Orchestrator.

For the July 14th update, R5012-20230327-GA-107522 is replaced by version R5014-20230713-GA as the required Edge software version. A customer can also upgrade their Edge to any Release 5.2.x Edge build.
Note: The Firmware Version parameters cannot be seen by some users due to their role and permissions (For example: Partner Administrators). If you cannot see these parameters and need to confirm them, please contact VMware Support for confirmation.
  1. After upgrading to R5014-20230713-GA, the Edge 6x0 platform version must show a status of HASupported Upgradable. This status can be seen when looking at Edge details on Monitor > Edge > Overview for that Edge.

    Note: This status may not show prior to the software upgrade to 5.0.1.2 depending on the Edge software being used prior to the upgrade, so it is important to check this status after upgrading to R5014-20230713-GA.

    Under the dropdown status box locate the firmware version and look for the HASupported Upgradable status.
 
 
Warning: If the Device Firmware > Platform Version status shows as Not Upgradable you will not be able to upgrade the Edge's Platform Firmware through the Orchestrator and the upgrade can only be done via CLI by Technical Support. An attempt to upgrade an Edge's Platform Firmware through the Orchestrator with a Not Upgradable status may result in the Edge becoming non-operational and requiring both a factory reset followed by a reactivation of the Edge.
 

 

  1. If an Edge upgraded to version R5014-20230713-GA shows as HASupported Upgradable, only then is the Edge upgraded to Platform Firmware Version R131-20221216-GA through the Orchestrator UI.
    1. An Operator Profile is used that has the R131-20221216-GA version listed for Platform Firmware.
    2. The Operator Profile includes not only the listed Platform Firmware but also Software Version R5014-20230713-GA. There is no Factory Version.
    3. The Edge Operations Team created and added this Operator Profile to all hosted Orchestrators running Orchestrator Release 5.x or higher.
Caution: While the firmware Operator Profile does include an Edge Software version R5014-20230713-GA, it is NOT to be used to upgrade the Edge's software. Upgrading the Edge software needs to be done as a separate step prior to upgrading the Edge firmware as outlined in Step 1 above.

Note: The Platform Firmware upgrade takes at least 10 minutes to complete and includes multiple Edge reboots where the Edge will show as offline. 
  1. Progress for Platform Firmware upgrades can be checked on the Orchestrator's Events section as you would for an Edge software upgrade.
    1. On the Classic Orchestrator:
  1. On the New Orchestrator a user can filter for "1.3.1" to see only those events, which is especially helpful when upgrading multiple Edges:
  1. Confirm the upgrade was successful by checking the Edge's drop-down status box on the Monitor > Edges page.
    1. On the Classic Orchestrator: 
    2. On the New Orchestrator:​​​​
  2. Once the Platform Firmware upgrade to 1.3.1 (R131-20221216-GA) is confirmed as successful, a user can either:
    1. Change the Operator Profile to R5014-20230713-GA, which was used in step one.
    2. If the user prefers to keep the Edge on a lower software release (for example, Release 4.3.1, or 4.5.1), the customer can temporarily upgrade the Edge to R5014-20230713-GA, perform the Platform Firmware upgrade to version 1.3.1 (R131-20221216-GA) so that the PIC version is v20N, and then downgrade the Edge’s software back to their preferred version. Downgrading the 6x0 Edge's software to an earlier version does not also downgrade the Edge's Platform Firmware and the Edge would continue to use Platform Firmware version 1.3.1 (R131-20221216-GA). In this use case the customer Edges would need to be on an Orchestrator using Release 5.x.

High Availability Upgrade Process

Beginning with Edge build R5014-20230713-GA upgrading an HA site to Platform Version R131-20221216-GA is the same as upgrading a single Edge standalone site. The software is upgraded normally (first the Standby is upgraded, reboots and then fails over to become the Active while the previous Active Edge upgrades) and then the Platform Firmware is separately upgraded in the same fashion. The entire process will take approximately 20-30 minutes:

  1. Upgrade the HA Edges to the Edge R5014-20230713-GA version as normal and confirm both Edges have the new build.
  2. Upgrade the HA Edges to the R131-20221216-GA Platform Version.
  3. Check the Edge Overview and confirm that both Edges have the 1.3.1 (R131-20221216-GA) Platform Version.
  4. Once the Platform Firmware upgrade to 1.3.1 (R131-20221216-GA) is confirmed as successful, a user can either:
    1. Change the Operator Profile to R5014-20230713-GA, which was the one used in step one.
    2. Or alternately, If the user prefers to keep the Edge on a lower software release (for example, Release 4.3.1, or 4.5.1), the customer can temporarily upgrade the Edge to R5014-20230713-GA, perform the Platform Firmware upgrade to version 1.3.1 (R131-20221216-GA) so that the PIC version is v20N, and then downgrade the Edge’s software back to their preferred version. Downgrading the 6x0 Edge's software to an earlier version does not also downgrade the Edge's Platform Firmware and the Edge would continue to use Platform Firmware version 1.3.1 (R131-20221216-GA). In this use case the customer Edges would need to be on an Orchestrator using Release 5.x.
Note: As of the April 4th, 2023 article update, Edge build R5014-20230713-GA is the only release where the HA upgrade process for Platform Firmware is automated through the Orchestrator.

Note: A previous version of this KB article also listed upgrading the Factory Image to a 5.x version as a required step. This was not correct, the Edge's factory version is not relevant to this process.

If you cannot upgrade your Edge to a 5.0.x build yet, see the Workaround section.
 



Workaround:

Ensure the power to the Edge is consistent and does not flap rapidly or consistently. A good example is a UPS to ensure a reliable power source.

To recover the Edge from the problem state:
  1. Disconnect the Edge from the power source.
  2. Wait 20 seconds
  3. Reconnect the Edge
If the issue recurs persistently and an upgrade to 5.0.1 is not yet possible, you can follow the same process as seen above for the upgrade, and once the process completes, you can change the operator profile back to what it was originally. The edge will be back to the original release and the PIC firmware will remain upgraded.
If you require assistance at any point please contact VMware Support and note this Article ID (88970) in the problem description. The support team can help manually upgrade the firmware.
For more information, see VMware SD-WAN – Support



Additional Information

Impact/Risks:

Note: The Firmware upgrade applied by support personnel as a workaround will be disruptive, as the Edge Services need to be shut down during the procedure. It is recommended to schedule a maintenance window. It also requires a local technician to connect to the Edge, as remote SSH access is lost during the process.

To avoid these risks, it is recommended to upgrade to the fixed version instead of applying the workaround.