Troubleshooting steps to follow when an SFP interface is not coming up on the VMware SD-WAN Edge

Products

• VMware SD-WAN by VeloCloud

Issue/Introduction

Symptoms:

• SFPx port not coming up on the VMware SD-WAN Edge.
• SFPx port not coming up after the peer device interface is shut and then unshut.

Environment

VMware SD-WAN by VeloCloud

Resolution

Best Practices for Connecting a new SFP Module to a VMWare SD-WAN Edge

​​​​​​To prevent the more common issues encountered when bringing up a new SFP / SFP+ module on an Edge, the following process is recommended when connecting an SFP module:

1. Please make sure the SFP module installed is supported for the platform you are using.
2. An SFP (or SFP+) module should be inserted into the Edge before the Edge is powered on. Additionally, to be safe, a valid link (fiber or Ethernet cable terminated into a live port at the remote end) should be applied to the SFP before powering on the Edge.
3. Additionally, the link should be set at the remote end to the desired speed, with Auto-Negotiation turned OFF.
4. Identify the speed and type of interface that is desired to be serviced by the SFP port of the Edge. For example, 1Gb short-reach on multimode fiber.
5. Choose the appropriate SFP from the supported list to match the remote link. For example, if the remote link is 1Gb short-reach on multimode fiber, select a multi-rate SX/SR module that supports a 1Gb connection, such as the Dell P/N “Y3KJN”, and insert the module then the link (fiber) into the Edge.
6. Make sure the remote end (switch or router) is set to 1Gb speed, and Auto-Negotiation (AN) is OFF, and the port is enabled (turned on)
7. At this point you can power on the Edge.
8. Check the physical connectivity of the SFP module with the Edge SFP interface.
9. Ensure the port is unshut on both ends.
10. Once the Edge fully boots, the link LED on the port should be amber for 1Gb interfaces and green for 10Gb interfaces. If the link light does not illuminate, there’s a problem with the link (refer to the next section on debugging).

Basic Troubleshooting Steps If There Is an Issue

1. In some cases, auto-negotiation may not work properly with some SFP models or modules made by some vendors due to interoperability issues. This is true of the 5x0 (520 and 540) and especially the 6x0 (610, 620, 640, and 680) Edge model lines. In those instances, you will need to configure the Speed/Duplex manually on the peer side.  For more information about the Edge 6x0 SFP issues, please consult the KB article SFP Port Auto-Negotiation Limitations on VMware SD-WAN Edge 6x0 Models (84474)

 

2. The SFP port may not come up when one side is using a 1G SFP and other side is using a 10G SFP. The recommendation is to use the same speed SFP on both sides. If you are using a 1G SFP on the Edge, please use a 1G SFP on the peer side as well (same applies when using a 10G SFP on the Edge). 
 

3. The Interface Speed/Duplex status can be verified on the Orchestrator.  Go to Remote Diagnostics and select an Edge.  Run the diagnostic Interface Status as shown below:

4. There is a known issue when connecting an Edge 840 to a Cisco C2960 switch using the SFP module Vardata VELO-1GE-LX-VAR.  In this configuration, the SFP interface will not come up on the Edge 840.  This same Cisco interface comes up as expected when connecting to an Edge 540.

• The Cisco switch tries to negotiate the speed at 10G as the Edge 840 is capable of operating at 10G speed, but the Cisco switch is never successful so the port remains down.  While Support first encountered this issue on a Cisco C2960 switch, this issue may be encountered on other Cisco switches and routers.
• To resolve this issue, the following items need to be configured on the Cisco device:

1. The Cisco device must be be configured to enable service for unsupported SFPs or transceivers. 
2. The Cisco device must have negotiation disabled for the SFP interface.  Example: using the 'speed nonegotiate' command for that interface.
3. Note: the commands to configure a Cisco device for these two items may vary depending on the vendor, device model, and OS version.  Please refer to the proper documentation to confirm the correct command for each item.

 

Advanced troubleshooting steps

1. If a multi-rate module is used, please ensure that the other end of the link is set to the desired speed and that Auto-Negotiation is disabled. 
2. A 10Gb optical module cannot be used in an Edge 520, 540, or  610, as these Edges only support 1Gb interfaces. 
3. Please confirm that the correct type of fiber-optic cable is being used. Single mode (often yellow) for the long reach (LR/LX) modules, and multi-mode (often orange or aqua) for short reach (SR/SX) modules.
4. Please ensure the Edge is using the most current software. At least one issue has been found in DPDK on Edge 6x0 models, that requires the Edge to be upgraded to the 3.4.1 Release. 
5. A copper (1000bT) module cannot be used on the 5x0, 6x0, or 840 systems in a High Availability topology.  The reason being that these Edge models do not implement RxLOS correctly, meaning that the link will always appear to be up even if there’s no cable plugged into the SFP.
6. We only support 1G and 10G modes. (We don't support 100Mbps or any other.)
7. Do Not Hot-Swap SFPs. This is not supported. If an SFP is changed, the system will need to be rebooted. In some cases a reboot may not be enough and a power cycle will need to be performed.  

Additional Notes

Optical SFPs have no auto-negotiation. The module takes the electrical signal and turns the light on or off, and vice versa. Most 10G (SFP+) opticals should be able to run even slower than 1G speed. In fact, many 10G SFP+ should work even in an Edge 520/540 model.

Some SFP and SFP+ modules may not work because they supply bad data in their SFF headers (the standardized header for SFPs) from the EEPROM on the module.

Some MACs (MAC in this case is the logical part of the overall SFP interface) can determine the speed of a signal coming in, and switch to either 1G or 10G speed. However, if both ends are manually configured to a fixed speed and the speed of each end is different (one end 1G, other end 10G), then no communication will occur. As a result, if there is no knowledge about the speed of the peer side, then there is no guarantee that the connection will come up.