How to Non-Disruptively Change the VLAN/IP for vSAN VMk in a Production Environment
Article ID: 326852
Updated On:
Products
VMware vSAN
Issue/Introduction
How to move/redirect data traffic in a production vSAN cluster to a new VLAN/subnet, without risking production interruption or requiring an outage to complete. (Action plan to re-IP the vSAN)
Note: Having cluster only partially configured with air-gap (two vSAN VMK interfaces "up" and connected to 2 different VLANs on some nodes, while other nodes are using only single vSAN VMK interface) is NOT a long-term supported solution.
Note: vSAN does not support multiple VSAN tagged vmkernel ports on the same subnet.
Please refer to the following VMWare documentation for vSAN networking best practices: vSAN Planning and Deployment
Environment
VMware vSAN(all versions)
Cause
- Occasionally, it is necessary to change the IP scheme in use for parts or all of the environment.
- A "VMK" HA functionality feature was introduced in vSAN which allows 2 VMK interfaces to have vSAN enabled on them at the same time.
- This KB takes you through the process step-by-step and assumes the original (old VLAN/subnet) VMK interface will be deleted.
Resolution
The steps outlined in this KB Article assume the following,
- The vSAN Cluster is currently serving production data.
- The expectation is to have vSAN traffic flowing on a new VLAN/subnet at the conclusion of this KB without disruption to production VMs.
- The vSAN hosts are already connected to 10G switch ports configured with the old/existing VLAN/subnet.
- The vSAN hosts are already connected to 10G switch ports configured with the new VLAN/subnet.
- A new standard vSwitch has been added with the new VLAN/subnet configured, or
- A new port group has been added to an existing vDS with the new VLAN/subnet configured.
/VSAN/IgnoreClusterMemberListupdatesis set to the default value of 0. If this setting is not 0, vCenter will not be able to push out any updates to the unicastagent list on the ESXi hosts resulting in a production outage.- Run the following command on all hosts in the cluster to check the current setting:
esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListupdates - If the value is 1 please set it back to the default value of 0 via the following command:
esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates
Steps required
- First, we need to list the VMKs currently configured.
- In this sample configuration, we have 4 hosts with IPs ending in -- .81 .82 .83 and .84
- In this sample configuration, we have 4 hosts with IPs ending in -- .81 .82 .83 and .84
- SSH to each host in your vSAN Cluster, login as root and type the following command:
esxcli network ip interface ipv4 get
You'll see output similar to below:Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS---- ------------- ------------- -------------- ------------ ----------- --------vmk0 192.###.###.81 255.###.######.######.###vmk1 192.###.######.######.###vmk2 192.###.######.######.### - Next, we need to know the VMK currently being used by vSAN:
esxcli vsan network list
Interface VmkNic Name: vmk2 <<----This is the VMK interface used for VSAN IP Protocol: IP Interface UUID: ########-####-####-####-############ Agent Group Multicast Address: 224.2.3.4 Agent Group IPv6 Multicast Address: 2001:db8::2:3:4 Agent Group Multicast Port: 23451 Master Group Multicast Address: 224.1.2.3 Master Group IPv6 Multicast Address: 2001:db8::1:2:3 Master Group Multicast Port: 12345 Host Unicast Channel Bound Port: 12321 Multicast TTL: 5 Traffic Type: vsan- To move the vSAN Data to the new VLAN, follow these steps:
- In the vSphere Web Client, create a new VMkernel Interface on each vSAN host - for this KB the new interface is: vmk3
- Configure the IP Addresses from your new VLAN on these new VMKs for each host, being sure to select the correct vSwitch and/or Portgroup.
- Enable vSAN on the new VMKs for each host
- You should now see the new VMK from the CLI with the following command:
esxcli network ip interface ipv4 get
You'll see output similar to below.
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS---- ------------- ------------- -------------- ------------ ----------- --------vmk0 192.###.######.######.######.###vmk1 192.###.######.######.###vmk2 192.###.######.######.###vmk3 192.###.######.######.###
- Now check to make sure vSAN is aware that both VMKs are tagged for vSAN traffic:
esxcli vsan network list - You should now see output listing two VMKs similar to below.
InterfaceVmkNic Name: vmk2 <<----Original vSAN VMK on Old VLANIP Protocol: IPInterface UUID: ########-####-####-####-############Agent Group Multicast Address: 224.2.3.4Agent Group IPv6 Multicast Address: 2001:db8::2:3:4Agent Group Multicast Port: 23451Master Group Multicast Address: 224.1.2.3Master Group IPv6 Multicast Address: 2001:db8::1:2:3Master Group Multicast Port: 12345Host Unicast Channel Bound Port: 12321Multicast TTL: 5Traffic Type: vsan <<----Tagged for vSAN traffic
InterfaceVmkNic Name: vmk3 <<----New vSAN VMK configured on the new VLANIP Protocol: IPInterface UUID: ########-####-####-####-############Agent Group Multicast Address: 224.2.3.4Agent Group IPv6 Multicast Address: 2001:db8::2:3:4Agent Group Multicast Port: 23451Master Group Multicast Address: 224.1.2.3Master Group IPv6 Multicast Address: 2001:db8::1:2:3Master Group Multicast Port: 12345Host Unicast Channel Bound Port: 12321Multicast TTL: 5Traffic Type: vsan <<----Tagged for vSAN traffic
- Next, we will test connectivity between all hosts on the new VMK interface (in this example it's vmk3).
This command will use the new VMK interface (vmk3) to ping the IP addresses of vmk3 on the other host.- If using MTU 1500:
vmkping -I vmk3 -s 1472 -d 192.###.###vmkping -I vmk3 -s 1472 -d 192.###.###vmkping -I vmk3 -s 1472 -d 192.###.### - If using MTU 9000(Jumbo frames):
vmkping -I vmk3 -s 8972 -d 192.###.###vmkping -I vmk3 -s 8972 -d 192.###.###vmkping -I vmk3 -s 8972 -d 192.###.###
Output will look similar to this on each ping:64 bytes from 192.###.###64 bytes from 192.###.###64 bytes from 192.###.###--- ping statistics ---3 packets transmitted, 3 packets received, +1 duplicates, 0% packet lossround-trip min/avg/max = 0.735/1.204/1.035 ms
- If using MTU 1500:
- When all hosts can ping each other on the new VMK interface, starting with one host:
- Go to the vSphere Web Client
- Highlight the host in the Inventory Pane
- Click the Configure Tab
- Under "Networking" highlight VMkernel Adapters
- Choose the old vmk (vmk2 in this sample) and click Edit (pencil icon)
- Remove the check mark next to vSAN - this disables vSAN traffic on the original VMK
- Verify all VMs are in an accessible state - this is precautionary, the expectation is that no VMs will be affected.
- If VMs start becoming non-compliant, shutting down, inaccessible, etc.
- Edit the old VMK and check the vSAN box
- Everything should recover immediately after turning vSAN back on for the original VMK interface.
- When the original VMK on all hosts has had vSAN disabled (check mark removed in edit mode) and all VMs/hosts are still compliant/operating as expected....
- Delete the old VMK Interface**
- Delete the old VMK Interface**
Additional Information
- If the steps in this KB Article are followed precisely, there is no risk to network disruption or data availability.
- The change is designed to be seamless, without impact to running Virtual Machines.
- As a precaution however, this KB advises disabling vSAN on the original VMK interface one host at a time, then briefly monitoring for evidence of VMs becoming non-compliant, shutting down, or becoming Inaccessible, etc. - again this should not happen.
- For a more cautious approach, when it's time to disable vSAN on the original VMK interfaces, you can place the host in maintenance mode, disable the original VMK, re-test ping connectivity to all hosts on the new VLAN, then exit maintenance mode and repeat for the next host.
- 本番環境における vSAN VMkernel ポートの VLAN/IP を無停止で変更する方法
Feedback
Yes
No
Powered by
