gateway to edge path instability due to rmsg corruption between 4.5 edge and 5.x gateway

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

The edge to gateway path can be seen unstable. On checking, Unstable and Unusable holdoff seconds and loss percentage can be abnormally high

 

• From edge to gateway the path is seen to be unstable--> This can be checked from "List paths" in remote diagnostics

GE4         NONE       NONE  gateway-1 14.18.189.104   121.35.254.51        DEFAULT  1578689614      ACTIVE  UNSTABLE  UNSTABLE             1             0       0.0       0.0     0.0     0.0          250000          250000       250000       250000    250000                  500  1500       No  OUT         0         1             0
GE2         NONE       NONE  gateway-1   58.251.94.11   121.35.254.51        DEFAULT  3946497606      ACTIVE  UNSTABLE  UNSTABLE             3             3       0.0       0.0     0.0     0.0          250000          250000       250000       250000    250000                  500  1500       No  OUT         0         1             0
GE5         NONE       NONE  gateway-1  10.122.10.237  10.122.200.254        DEFAULT  2736182472      ACTIVE  UNSTABLE  UNSTABLE             2             1       0.0       0.0     0.0     0.0          250000          250000       250000       250000    250000                  500  1500       No  OUT         0         1             1

• From gateway to edge we can see the same path to be stable
• path_unstable_loss_pct and unusableLossThresholdPct shows a high junk value
   
  

   "pathState": "ACTIVE",
          "pathStateRx": "UNSTABLE",
          "pathStateTx": "UNSTABLE",
          "pathUpMs": 179180670,
          "peerLinkWanSg": 135,
          "prio8021P": 0,
          "remoteRx": 30000,
          "secureHB": 257,
          "serial": 752818907,
          "transientPath": 0,
          "tunnelInitiator": 0,
          "unstableHoldoffMs": 60000,
          "unstableLossThresholdPct": 1090802244,
          "unusableLossThresholdPct": 1145122126,

 

Cause

Due to an rmsg corruption between 4.5 edge and 5.x gateway, abnormally high junk values can be set for the vcmp holdoff seconds and loss percentage

As a preventive fix, the path thresholds are sanitized before assignment.

Resolution

As part of preventive HF on gw over 5.0.1.3, we can do a sanity check before upgrading the path threshold values and wont set them into these high values. This would solve the path threshold issue

Issue is fixed as a part of #139936