The steps outlined in this KB Article assume the following,
1. The vSAN Cluster is currently serving production data
2. The expectation is to have vSAN traffic flowing on a new VLAN at the conclusion of this KB without disruption to production VMs.
3. The vSAN hosts are already connected to 10G switch ports configured with the old/existing VLAN
4. The vSAN hosts are already connected to 10G switch ports configured with the new VLAN.
5. A New Standard vSwitch has been Added with the NEW VLAN configured, OR
6. A new port group has been added to an existing vDS with the new VLAN configured on it.
7. /VSAN/IgnoreClusterMemberListupdates is set to the default value of 0. If this setting is not 0 vCenter will not be able to push out any updates to the unicastagent list on the ESXi hosts resulting in a production outage.
Run the command esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListupdates on all hosts in the cluster to check the current setting.
If the value is 1 please set it back to the default value of 0 via the following command:
esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates
+-------------------------------------------------------+
Steps required
First, we need to list the VMks currently configured.
In this sample configuration, we have 4 hosts with IPs ending in -- .81 .82 .83 and .84
**SSH to each host in your vSAN Cluster, login as root and type the following command
# esxcli network ip interface ipv4 get
You'll see output similar to below.
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
---- ------------- ------------- -------------- ------------ ----------- --------
vmk0 192.0.0.81 255.255.255.0 192.0.0.255 STATIC 192.0.0.1 false
vmk1 192.0.10.81 255.255.255.0 192.0.10.255 STATIC 0.0.0.0 false <<----VLAN10
vmk2 192.0.20.81 255.255.255.0 192.0.20.255 STATIC 0.0.0.0 false <<----VLAN20
Next, we need to know the VMK currently being used by vSAN with this command.
# esxcli vsan network list
You'll see,
# esxcli vsan network list
Interface
VmkNic Name: vmk2 <<----This is the VMK interface used for VSAN
IP Protocol: IP
Interface UUID: ########-####-####-####-########6fbb
Agent Group Multicast Address: ###.2.3.4
Agent Group IPv6 Multicast Address: ####::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: ###.1.2.3
Master Group IPv6 Multicast Address: ####::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan
To move the vSAN Data to the new VLAN, follow these steps:
1. In the vSphere Web Client, create a new VMkernel Interface on each vSAN host - for this KB the new interface is: vmk3
2. Configure the IP Addresses from your new VLAN on these new VMKs for each host, being sure to select the correct vSwitch and/or Portgroup.
3. Enable vSAN on the new VMKs for each host
4. You should now see the new VMK from the CLI with the following command:
# esxcli network ip interface ipv4 get
You'll see output similar to below.
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
---- ------------- ------------- -------------- ------------ ----------- --------
vmk0 192.0.0.81 255.255.255.0 192.0.0.255 STATIC 192.0.0.1 false
vmk1 192.0.10.81 255.255.255.0 192.0.10.255 STATIC 0.0.0.0 false <<----VLAN10
vmk2 192.0.20.81 255.255.255.0 192.0.20.255 STATIC 0.0.0.0 false <<----VLAN20
vmk3 192.0.30.81 255.255.255.0 192.0.30.255 STATIC 0.0.0.0 false <<----New VMK on VLAN30
5. Now check to make sure vSAN is aware that both VMKs are tagged for vSAN traffic.
# esxcli vsan network list
You should now see output listing two VMKs similar to below.
# esxcli vsan network list
Interface
VmkNic Name: vmk2 <<----Original vSAN VMK on Old VLAN
IP Protocol: IP
Interface UUID: ########-####-####-####-########6fbb
Agent Group Multicast Address: ###.2.3.4
Agent Group IPv6 Multicast Address: ####::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: ###.1.2.3
Master Group IPv6 Multicast Address: ####::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan <<----Tagged for vSAN traffic
Interface
VmkNic Name: vmk3 <<----New vSAN VMK configured on the new VLAN
IP Protocol: IP
Interface UUID: ########-####-####-####-########6fbb
Agent Group Multicast Address: ###.2.3.4
Agent Group IPv6 Multicast Address: ####::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: ###.1.2.3
Master Group IPv6 Multicast Address: ####::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan <<----Tagged for vSAN traffic
+---------------------------------------------------+
6. Next, we will test connectivity between all hosts on the new VMK interface (in this example it's vmk3)
This command will use the new VMK interface (vmk3) to ping the IP addresses of vmk3 on the other host.
If using MTU 1500
# vmkping -I vmk3 -s 1472 -d 192.0.30.82
# vmkping -I vmk3 -s 1472 -d 192.0.30.83
# vmkping -I vmk3 -s 1472 -d 192.0.30.84
If using MTU 9000(Jumbo frames)
# vmkping -I vmk3 -s 8972 -d 192.0.30.82
# vmkping -I vmk3 -s 8972 -d 192.0.30.83
# vmkping -I vmk3 -s 8972 -d 192.0.30.84
Output will look similar to this on each ping.
64 bytes from 192.168.30.82: icmp_seq=0 ttl=64 time=0.933 ms
64 bytes from 192.168.30.83: icmp_seq=1 ttl=64 time=0.908 ms
64 bytes from 192.168.30.84: icmp_seq=2 ttl=64 time=0.735 ms
--- ping statistics ---
3 packets transmitted, 3 packets received, +1 duplicates, 0% packet loss
round-trip min/avg/max = 0.735/1.204/1.035 ms
7. When all hosts can ping each other on the new VMK interface, starting with one host.
**Go to the vSphere Web Client
**Highlight the host in the Inventory Pane
**Click the Configure Tab
**Under "Networking" highlight VMkernel Adapters
**Choose the old vmk (vmk2 in this sample) and click Edit (pencil icon)
**Remove the check mark next to vSAN - this disables vSAN traffic on the original VMK
8. Verify all VMs are in an accessible state - this is precautionary, the expectation is that no VMs will be affected.
**If VMs start becoming non-compliant, shutting down, inaccessible, etc.
**Edit the old VMK and check the vSAN box
**Everything should recover immediately after turning vSAN back on for the original VMK interface.
9. When the original VMK on all hosts has had vSAN disabled (check mark removed in edit mode) and all VMs/hosts are still compliant/operating as expected....
**Delete the old VMK Interface**