Isolating the Network Traffic of vSphere Replication
Article ID: 312753
Updated On:
Products
Issue/Introduction
This article will walk you through the procedure of isolating vSphere Replication traffic step by step.
You can isolate the network traffic of vSphere Replication from all other traffic in a datacentre’s network.
Isolating the replication traffic helps you ensure that sensitive information is not routed to the wrong destination. It also helps you enhance the network performance in the data center, because the traffic that vSphere Replication generates does not impact other types of traffic. Traffic isolation also facilitates monitoring and troubleshooting. You isolate the network traffic to the vSphere Replication Server by dedicating a VMkernel NIC on each ESXi host on the primary site that sends data to the vSphere Replication Server.
If you are using a distributed network switch, you can take advantage of the vSphere Network I/O Control feature to set limits or shares for incoming and outgoing replication traffic on each ESXi host. The feature allows you to manage the network resources that vSphere Replication uses.
By default, the vSphere Replication appliance has one VM network adapter that is used for various traffic types.
- Management traffic between vSphere Replication Management Server and vSphere Replication Server.
- Replication traffic from the source ESXi hosts to the vSphere Replication Server.
- Traffic between vCenter Server and vSphere Replication Management Server.
- NFC (Network File Copy) traffic, which is the traffic from the vSphere Replication Server appliance at the target site to the destination datastores.
You can add network adapters to the vSphere Replication appliance and use the VAMI to configure a separate IP address to use for each traffic type.
You can isolate the vSphere Replication NFC traffic from the vSphere Replication Server to the target datastore. By default, the NFC traffic is sent to the target ESXi host from the vSphere Replication Server through the management network. You can isolate the NFC traffic from the management traffic by sending it through the replication network. In this case, the vSphere Replication server will handle the replication and NFC traffic together, by using the same interface. To isolate the replication and NFC traffic from the management traffic, you must add a second vNIC to separate them. Alternatively, you can add a third vNIC for NFC traffic only. This option provides security isolation with a dedicated vSphere Replication VLAN for replication traffic and another one for NFC traffic, depending on the security requirements in your environment.
NOTE:
- One VMkernel adapter must handle one traffic type to get the best results.
- vSphere Replication requires that all components in your environment, such as vCenter Server, ESXi hosts, and the vSphere Replication appliance use the same IP version, IPv4 or IPv6.
- You can use VLANs to separate traffic types
- After the IP address of the vSphere Replication server on the target site changes, you must manually reconfigure replications on the source site to point to the new IP address.
- Distributed switches can be used with NIOC
You may be required to separate replication traffic based on security or performance needs. Also, the replication traffic will consume a lot of network bandwidth in your infrastructure limiting or constraining the available bandwidth of other applications running on the management network affecting overall performance.
Isolating the replication traffic helps you ensure that sensitive information is not routed to the wrong destination. It also helps you enhance the network performance in the data center, because the traffic that vSphere Replication generates does not impact other types of traffic.
- By default, VRMS has 1 vNIC that is used for various traffic types
- Management traffic between VRMS and VR Server
- Replication traffic from the source ESXi hosts to the VR Server
- Traffic between vCenter Server and VRMS
- NFC (Network File Copy) traffic, which is the traffic from the vSphere Replication Server appliance at the target site to the destination datastores.
You can add network adapters to the vSphere Replication appliance and use the VAMI to configure a separate IP address to use for each traffic type. We will be isolating VR & VR NFC traffic from the management network.
The picture below shows replication traffic flowing in a single direction.
The picture below shows how Management, Replication & NFC traffic flows can be isolated over separate networks.
We will be using the IP scheme referenced below for configuring the different networks
MGMT - Management traffic
VR - vSphere Replication traffic
VR NFC - vSphere Replication NFC traffic
Isolating the Network Traffic of vSphere Replication - Isolating the Network Traffic of vSphere Replication. Refer to this guide for more information before proceeding further.
Set up a VMkernel Adapter for VR & VR NFC Traffic on Source & Target hosts & enable the relevant services on them
Configure virtual machine port groups for the respective VMkernel adapters on all the hosts that will be used for replication.
Note : From the picture above you can see that I have created 2 VMkernel adapters and 2 Virtual Machine Port Groups. VM Portgroups are VR & VR-NFC. VMKernel adapters are vSphere Replication & vSphere Replication NFC
Configure vSphere Replication NICs
Configure vSphere replication network
To re-direct traffic to source and target hosts, you need to assign the correct networks to the VR appliance and a static route may be required if the source/target network is not properly routed.
Access the appliance by going to - https://VRA-IP:5480.
Go to Network > Address tab and specify the network parameters for the two added NICs:
eth1: Incoming traffic (from source hosts to appliance)
eth2: Outgoing traffic (from appliance to target hosts)
The image above is for representation only.
Go to VR > Configuration page and enter the IP address assigned to the eth1 network adapter in the IP address for Incoming Storage Traffic field. This tells the VR appliance incoming traffic (from source hosts) will arrive on eth1. Click Apply Network Settings to save the configuration.
Information from release notes:
An unexpected error appears in the Summary tab if you have a second NIC, configured with a static route set
If you have a second NIC and if you configured a static route set for it, the VRMS Appliance Management Interface cannot obtain an IP address from the second NIC. This might result in the following unexpected error in the IP Address for Incoming Storage Traffic field, under the Summary tab:
The Storage Traffic IP address <IP_address> must match one of the NIC IP addresses
Workaround: The NIC is correctly configured and you can discard the error.
You can verify if the replication traffic uses the correct IP address by running the following command:
cat /etc/vmware/hbrsrv-nic.xml
You may see the above error when you are configuring multiple NICs for traffic isolation, this is a know error documented in the release notes of all the VR versions and can be safely ignored. But, you will need to configure static routes for the traffic to flow correctly.
vSphere Replication 8.5 Release Notes
vSphere Replication 8.6 Release Notes
vSphere Replication 8.7 Release Notes
vSphere Replication 8.8 Release Notes
You will see this message under gateway in the VAMI - The configuration supports only one default gateway on the appliance. This will replace the existing default gateway and set the one you specify here as the global default gateway. The basic network configuration tools in Linux provide the ability to configure a maximum of one default gateway. However, with advanced routing techniques, specifically static routes, you can configure multiple gateways. Hence use one gateway for all your traffic types including Management, vSphere Replication & vSphere Replication NFC.
Note: The IP configuration via the VRMS Appliance Management Interface supports only one default gateway on the vSphere Replication appliance. To configure a static route on an additional NIC, see Configure a Static Route on an Additional VM Network Adapter
Adding static routes
If the appliance can't reach the source OR destination network (it depends where the appliance is placed), you need to add a static route. To add a static route to the vSphere Replication, you need to SSH the appliance. By default SSH service is disabled and you need to enable SSH on the VRA
Access the vSphere Replication's console via HTML client & login as root
From the console, run the following command to enable the SSH service in the appliance:
root@VRMSPR [ ~ ]# /usr/bin/enable-sshd.sh
root@VRMSPR [ ~ ]# service sshd start
Add the route details in the file below to make the change persistent across reboots.
root@VRMSPR [ ~ ]# cd /etc/systemd/network
root@VRMSPR [ /etc/systemd/network ]# ls -l
-rw-r--r-- 1 root root 119 Nov 13 19:20 10-eth0.network -> Management
-rw-r--r-- 1 root root 117 Nov 13 19:20 10-eth1.network -> VR Traffic
-rw-r--r-- 1 root root 117 Nov 13 19:20 10-eth2.network -> VR NFC Traffic
NOTE: Edit the NIC configured for VR traffic, this could depend on your NIC configuration in the appliance
NOTE: Upgrading vSphere Replication appliance erases the static routes saved in the network files. Please save the routing information of all the appliances on a notepad before upgrading the appliances and restore them after the upgrade is complete. Not doing this can trigger errors like -
A replication error occurred at the vSphere Replication Server for replication 'VM Name'. Details: 'No connection to VR Server for virtual machine VM Name on host ESXi.host.local in cluster (Cluster name) in (Datacenter name): Unknown'.
Synchronization monitoring has stopped. Please verify replication traffic connectivity between the source host and the target vSphere Replication Server. Synchronization monitoring will resume when connectivity issues are resolved.
Running tests to check if static route is working
To check if the configuration has been applied in the routing table of vSphere replication appliance, run the command:
# netstat –r
To verify the VR appliance & hosts are reachable from the source & target hosts bi-directionally, run the vmkping command
SSH into the source & target host and run this command to check if the hosts are able to reach the vSphere Replication appliance:
vmkping -I <VMK# of VR network> <IP address of VR appliance>
You can also PING the hosts from the VR appliance to test connectivity.
In addition, you must configure the relevant static routes on each ESXi host at the source site to communicate with the target site. For replications to flow in the opposite direction, you must configure reverse routes on the ESXi hosts on the target site. See https://knowledge.broadcom.com/external/article?legacyId=2001426. Depending on the complexity of your environment, if you want to isolate the NFC traffic, you must configure the relevant vSphere Replication and NFC vSphere Replication static routes after you configure the VMkernel adapters for vSphere Replication and NFC traffic.
We have successfully completed isolating VR & VR NFC network now. We can configure new replications or reconfigure existing replications for the traffic to flow on their configured networks.
Environment
Resolution
Feedback
