Realtime Business collaboration apps MS Teams / Zoom / Webex have poor Audio / Video quality

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

This article details the troubleshooting steps and the best practices to check and avoid common Audio / Video issues with Business collaboration apps such as  MS Teams / Zoom / Webex.

Environment

VMware Velocloud SD-WAN Edge
VMware Velocloud SD-WAN Edge with NSD via Edge / Gateway
VMware Velocloud SD-WAN Edge with CSS tunnels.

Cause

When there is a congestion in the WAN link usage i.e whenever traffic spikes up to the max WAN link(s) capacity, Realtime applications will receive their share of assured bandwidth from the scheduler based on the Priority configured.

VMware SD-WAN edge provides out of the box "application-aware QoS" that prioritizes Realtime applications.

Refer the below document to understand how VMware SD-WAN edge provides out of the box "application-aware QoS"

https://docs.vmware.com/en/VMware-SD-WAN/6.0/VMware-SD-WAN-Administration-Guide/GUID-4A7D7A8E-5537-47B4-A88D-6F881F377DA9.html?hWord=N4IghgNiBcIM4GMAWBTAJgVwgSwHYHMQBfIA

The below document shows the default rules that can be inherited from "quick start" profile if "quick start" profile or a derivative of "quick start" profile is used while creating a new edge.

https://docs.vmware.com/en/VMware-SD-WAN/6.0/VMware-SD-WAN-Administration-Guide/GUID-E3CD3284-5A67-4A60-B8AA-E5F811AAF5AF.html

As per these out of the box policies, MS Teams / Zoom / Webex applications that belong to "All Business collaboration" category are assigned with priority "High" and service class "Realtime" (Refer rule name "Business collaboration" in below image) ; however, when customers modify the business policy rules and rule order or add "edge override" business policy rules, configuration best practices to prioritize the real time apps has to be followed.











This article explains the troubleshooting steps to be followed in a scenario where customized business policy rules could be the reason for poor audio / video quality in Realtime applications such as MS Teams / Zoom / Webex.

Resolution

Verify the traffic classification:

> Open remote diagnostics for the edge, search the affected flow from "List active flows", identify the business policy applied for the flows related to MS Teams / Zoom / Webex application.

For example, MS Teams Audio / Video flows use UDP ports (3478 - 3481), under "List active flows", provide source IP address, provide destination port as anything between 3478 - 3481.

MS Teams IPs and ports: https://learn.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-worldwide




> Check the business policy configuration and identify the traffic classification for the business policy, the Priority has to be "High / Normal" and Service Class has to be "Realtime". 

Example:




> If the traffic classification / business policy matched is not correct, evaluate the rule order configured to understand the reason for not matching the default "business collaboration" policy. If the default "business collaboration" policy could not be matched due to rule order used, refer next section to setup an "Edge override" business policy rule for MS Teams / Zoom / Webex application.



Setup "Edge override" Business policy rule for Real time Internet hosted apps MS Teams / Zoom / Webex :

> Edge evaluates the matching Business policy for a new flow based on the order of business policy rules, it is necessary to position the rule for MS Teams / Zoom / Webex in such a way that none of the rules above the intended rule matches flows pertaining to MS Teams / Zoom / Webex.
> Business policy rules for Apps such as MS Teams / Zoom / Webex have to be configured with Priority "High / Normal" and Service Class "Realtime"
> Set up the source , destination match conditions under the match tab, choose the specific application under the application category on the business policy match condition, once the application is selected, verify the default service class chosen for the selected application in the action tab, it has to be "Realtime" for apps such as MS Teams / Zoom / Webex, set the "Network service" as required.

Note: If the default classification is not Realtime, the app-map associated with the operator profile / software image could be old, contact VMware support to fix the problem.

> Choose the priority for the application under the business policy action tab.

> Once the rule is created, use remote diagnostics to verify the business policy applied for the MS Teams / Zoom / Webex flows.

Check the link performance:

> If the Audio / Video slowness issues continue after verifying the business policy, check the SDWAN path used by the application under Edge monitor > paths tab, check the QOE score for Audio / Video traffic, check the recent statistics for Packet loss, Jitter, Latency. Involve ISP team if link level issues are seen.

Note: The above option is applicable only if the traffic is routed via one of the below SDWAN peers:

Primary VCG (if network service was set as "Multi-path")
NSD VCG (if network service was set as "Backhaul > {NSD via Gateway name}")
SDWAN Hub (if network service was set as "Backhaul > {Hub name}")

> If the Realtime application is routed via direct or CSS tunnels or NSD via Edge, check the WAN link performance by looking into the path statistics for Primary VCG (Closest VCG) similar to the above step, check for WAN link flaps, NSD via Edge / CSS tunnel flaps from edge events, collect and analyze a pcap on WAN interface to verify if the issue is due to ISP network.
> For NSD via Edge / CSS setup, confirm if the closest POP is used, involve the CSS / NSD provider for troubleshooting the issue if multiple edges with same POP are affected.
> Check the Interface drop counter increments, speed and duplex negotiation on all Interfaces in the path i.e from customer PC / Laptop to the SDWAN edge LAN, WAN interfaces.

Involve VMware SASE support if further assistance is required to troubleshoot the issue.