Troubleshooting HCX Mobility Optimized Networking (MON) Asymmetric Routing Issues

Products

• VMware HCX
• Azure VMware Solution

Issue/Introduction

HCX Mobility Optimized Networking (MON) can experience routing inconsistencies when deployed in Azure VMware Solution (AVS) environments. These issues commonly manifest as asymmetric routing between on-premises and cloud environments, particularly after enabling MON features. This article covers common scenarios, causes, and resolutions for MON-related routing issues.

Environment

HCX

Azure VMware Solution

ExpressRoute connection between on-premises and Azure

Network configurations involving:

• On-premises firewalls
• T0/T1 gateways
• Stretched networks
• BGP routing

Cause

Several factors can contribute to routing issues in MON environments

1. Static Route Conflicts
   • Scenario 1: Unnecessary MON Network Traversal
     • Static routes in on-premises or cloud environments can force traffic over the MON extended network unnecessarily
     • Traffic that should use direct paths (like ExpressRoute) gets forced through the extended network
     • Results in increased latency and unnecessary network overhead
     • Can cause performance degradation for applications that should use local routing
     • May impact bandwidth available for necessary MON traffic
   • Scenario 2: Prevented MON Network Usage
     • Route conflicts can prevent traffic from using the MON extended network when it should
     • Traffic intended to use the extended network gets routed through alternative paths
     • Can break application connectivity that requires L2 adjacency
     • May cause communication failures between resources that expect to be on the same network segment
     • Results in MON optimization features not being utilized as designed
2. Policy Route Misconfigurations
   • Default policy routes can cause asymmetric routing
   • Incorrect subnet matching in policy routes
   • RFC 1918 address handling conflicts
3. Gateway Location Issues
   • VM gateways not properly aligned with their respective environments (on-premises or cloud)
   • Policy routes are only evaluated when the VM gateway is correctly located for its environment
   • Misaligned gateway locations can cause traffic tromboning and increased latency
   • Migration of VM gateway to appropriate side is essential for optimal VM-to-VM communication
   • Without proper gateway alignment, traffic may follow suboptimal paths even with MON enabled
   • Local traffic between VMs may be forced to traverse unnecessarily between environments
   • Gateway location directly impacts the effectiveness of MON's traffic optimization capabilities
   • Improper gateway location can lead to increased latency and network overhead

Resolution

Route Configuration Optimization

• Remove conflicting static routes from both on-premises and cloud environments
• Remove any static routes in firewalls or edge devices that could cause tromboning
• Verify and clean up routes in both environments to ensure proper traffic flow
• Identify and correct routes that prevent intended MON network usage
• Remove routes that force unnecessary MON network traversal
• Implement proper route advertisement through BGP
• Ensure correct handling of /32 routes advertised over BGP to peers
• Configure appropriate policy routes based on network requirements
• Regular audit of routes on both sides to prevent reintroduction of conflicts
• Validate traffic paths after route changes to ensure proper flow

Policy Route Management

• Consider removing all default policy routes to prevent asymmetric traffic
• Only implement policy routes after thorough network infrastructure assessment
• Specifically address required subnets in policy routes
• Evaluate RFC 1918 traffic handling requirements

Gateway Migration

• Ensure VM gateways are migrated to their expected sides in the connection:
  • On-premises VMs should use on-premises gateways
  • Cloud VMs should use cloud-side gateways
• Verify proper policy route evaluation after gateway configuration
• Monitor traffic patterns to confirm optimal routing based on gateway placement
• Regularly validate gateway configurations match intended network design

Traffic Flow Optimization

• Carefully plan how VMs on NE MON segments will access the internet
• Choose between T0 gateway in Azure VMware Solution or NE back to on-premises
• Document and implement consistent traffic flow patterns
• Validate traffic paths meet intended design

Additional Information

Important Considerations

1. MON is not supported with third-party gateways. It must be used with the T1 gateway directly connected to the T0 gateway without an NVA.
2. Policy Route Behavior:
   • By default, all RFC 1918 IP addresses are included in MON policy routes
   • Policy routes are only evaluated if the VM gateway is migrated to the cloud
   • Matching subnets for destinations are tunneled over the NE appliance
   • Non-matching traffic is routed through the T0 gateway
3. Special Azure VMware Solution Considerations:
   • Pay attention to /32 routes advertised over BGP to peers
   • Consider impact on both on-premises and Azure over ExpressRoute connection
   • Be aware of potential impacts on stateful firewalls
   • Plan for internet access requirements for VMs on NE MON segments

Best Practices

1. Regular routing configuration review
2. Thorough testing of policy routes before implementation
3. Documentation of intended traffic flows
4. Validation of gateway locations and migration status
5. Careful consideration of internet access paths for MON-enabled segments
6. Regular validation of traffic paths against intended design

Please see this Microsoft Article for more information - VMware HCX Mobility Optimized Networking (MON) guidance