Traffic Not Following HCX Mobility Optimized Networking (MON) Policy Routes As Expected

Products

• VMware HCX

Issue/Introduction

After extending networks using HCX and enabling Mobility Optimized Networking (MON), traffic from virtual machines in the cloud is not staying within the cloud environment despite configuring deny rules in the MON Policy Routes. This causes asynchronous routing where traffic ingress and egress paths differ. The traffic continues to flow through the HCX tunnel to on-premises instead of being forwarded to the Tier-1/Tier-0 gateway as expected.

Key Concepts for Understanding MON Policy Routes

Important: MON Policy Routes evaluate only the destination IP address of packets from MON-enabled cloud VMs. This is critical to understand for proper traffic control. When we refer to "traffic" in this article, we specifically mean "traffic destined for a particular IP address or subnet."

How Policy Routes Work

1. Allow/Deny have counterintuitive meanings in MON Policy Routes:
   • Allow means "Send traffic destined for this IP/subnet to on-premises via the HCX tunnel"
   • Deny means "Keep traffic destined for this IP/subnet in the cloud using the local gateway"
2. Rule evaluation is based on specificity, not rule order:
   • Most specific routes (longer prefix) take precedence (e.g., /32 before /24 before /16)
   • If rules have the same specificity, the rule listed first in the configuration is applied
3. Default configuration routes most traffic to on-premises:
   • Default rules allow all RFC-1918 ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to go on-premises
   • To keep specific traffic in the cloud, you must add more specific deny rules
4. For sites with multiple Service Meshes:
   1. The Mon Site names will all show 'hcx-cloud' as that name is based on the service type.
   2. They are listed in the same order that the service Mesh's were created, so you may have to check the UI, and count in the list which service mesh you want to select the correct Mon site.

Traffic Flow with MON

When a VM with MON enabled sends packets:

• If the destination IP is within the same extended network: traffic flows directly at Layer 2
• If the destination IP is in another segment within the cloud's NSX-T environment: traffic uses the local Tier-1 router
• If the destination IP is outside the cloud NSX-T environment: Policy Routes are evaluated to determine path

Steps to validate:

• Traffic from VM in AVS with MON enabled is being sent through the HCX tunnel to on-premises despite deny rule configuration
• Packet capture shows asymmetric routing where ingress and egress paths differ
• Checking the route entries in the policy routes configuration shows the deny rule is not taking precedence over existing allow rules

Environment

• VMware HCX with Network Extension and MON enabled
• Extended network with VMs in the cloud environment

Cause

The issue occurs due to the unique way MON Policy Routes evaluate traffic and make routing decisions based on destination IP address specificity and allow/deny settings. Two main causes contribute to this problem:

Rule evaluation order

When multiple policy routes exist, they are evaluated according to their specificity (more specific routes take precedence), not the order in which they appear in the list. This means a more specific rule with a longer prefix (e.g., /32) will be evaluated before a less specific rule with a shorter prefix (e.g., /24), regardless of their position in the policy route list.

Allow vs. deny configuration

In MON Policy Routes, "Allow" and "Deny" have special meanings that are often counterintuitive:

• Allow means "Send traffic destined for this IP/subnet to on-premises via the HCX tunnel"
• Deny means "Keep traffic destined for this IP/subnet in the cloud using the local gateway"

This is opposite to what most network administrators expect, where typically "allow" permits traffic to flow somewhere and "deny" blocks it.

For example, consider a configuration where there is a broad subnet rule (172.16.0.0/12 set to Allow) and a more specific subnet rule (172.16.5.10/32 set to Deny). An administrator might expect traffic to 172.16.5.10 to stay in the cloud because of the specific deny rule, but depending on how the rules are evaluated, the traffic might still go to on-premises due to the broader allow rule taking precedence.

How Traffic Flow Actually Works

When a VM with MON enabled tries to send traffic:

1. If the destination IP is within the same extended network, traffic flows directly at Layer 2
2. If the destination IP is in another segment within the cloud's NSX-T environment, traffic uses the local Tier-1 router
3. If the destination IP is outside the cloud NSX-T environment, the Policy Routes are evaluated:
   • If the destination IP is matched by an Allow rule: Traffic goes back to on-premises via the HCX tunnel
   • If the destination IP is matched by a Deny rule: Traffic stays in the cloud and uses the Tier-1/Tier-0 gateway
   • If the destination IP is not matched by any rule: Traffic stays in the cloud and uses the Tier-1/Tier-0 gateway

Policy Route Evaluation Order

The order of evaluation is based on specificity, not the order in which rules appear in the interface:

• Most specific routes (smaller subnet/larger prefix) take precedence
• A /32 rule is more specific than a /24, which is more specific than a /16
• If rules have the same specificity, the rule listed first in the configuration is applied

Default Configuration

By default, HCX creates the following policy routes:

• All RFC-1918 ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) configured as Allow
• This means all private IP traffic is sent to on-premises by default
• Any non-matching traffic (including internet traffic) stays in the cloud

This default configuration can cause issues when you want specific RFC-1918 destination traffic to stay in the cloud, requiring you to add more specific Deny rules.

Resolution

To correctly configure MON Policy Routes so that traffic destined for specific IP addresses remains in the cloud rather than going on-premises:

Access the HCX Network Extension interface

1. Log in to HCX Manager via the HCX Center Plugin or direct URL
2. Navigate to Network Extension > Advanced > Policy Routes

Configure deny rules for destination IPs that should stay in the cloud

1. Add an explicit DENY rule for the specific subnet (e.g., ###.###.###.###/32) that should not go to the on-premises environment
2. Ensure this rule is more specific than any ALLOW rules that might include this IP range
3. Remember that Policy Routes only affect traffic when the VM gateway is migrated to the cloud

For overlapping subnets, use the most specific network mask

• If you need to deny a specific destination IP (e.g., ###.###.###.###/32) that is part of a larger network that is allowed (e.g., ###.###.###.###/12), the deny rule must use a more specific network mask (/32) to override the broader allow rule

Verify the configuration:

1. After applying the changes, check the traffic flow using tools like traceroute
2. Confirm that traffic destined for denied subnets stays within the cloud environment
3. Verify that traffic to other destination IPs still flows to on-premises as expected

For comprehensive control

1. Consider removing the default RFC-1918 entries and adding specific allow/deny rules based on exact traffic flow requirements
2. Add explicit allow rules only for the specific destination subnets that need to go on-premises
3. Add explicit deny rules for destination subnets that should stay in the cloud

Understanding this proper configuration ensures traffic to specific IP ranges remains in the cloud, preventing asynchronous routing issues and optimizing network traffic flows.

If the error persists after following these steps, contact Broadcom Support for further assistance.

Additional Information

• VMware HCX Mobility Optimized Networking (MON) guidance - Azure VMware Solution
• Mobility Optimized Networking Policy Routes
• Uploading files to cases on the Broadcom Support Portal
• Collecting diagnostic information for VMware HCX