How to Non-Disruptively Change the VLAN/IP for vSAN VMk in a Production Environment
search cancel

How to Non-Disruptively Change the VLAN/IP for vSAN VMk in a Production Environment

book

Article ID: 326852

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

How to move/redirect data traffic in a production vSAN cluster to a new VLAN/subnet, without risking production interruption or requiring an outage to complete.

Note: Having cluster only partially configured with air-gap (two vSAN VMK interfaces "up" and connected to 2 different VLANs on some nodes, while other nodes are using only single vSAN VMK interface) is NOT a long-term supported solution.  
Note: vSAN does not support multiple VSAN tagged vmkernel ports on the same subnet.
 
Please refer to the following VMWare documentation for vSAN networking best practices: vSAN Planning and Deployment
 

Environment

7.x, 8.x

Cause

  • Occasionally, it is necessary to change the IP scheme in use for parts or all of the environment.
  • A "VMK" HA functionality feature was introduced in vSAN which allows 2 VMK interfaces to have vSAN enabled on them at the same time.
  • This KB takes you through the process step-by-step and assumes the original (old VLAN/subnet) VMK interface will be deleted.

Resolution

The steps outlined in this KB Article assume the following,
  • The vSAN Cluster is currently serving production data.
  • The expectation is to have vSAN traffic flowing on a new VLAN/subnet at the conclusion of this KB without disruption to production VMs.
  • The vSAN hosts are already connected to 10G switch ports configured with the old/existing VLAN/subnet.
  • The vSAN hosts are already connected to 10G switch ports configured with the new VLAN/subnet.
  • A new standard vSwitch has been added with the new VLAN/subnet configured, or
  • A new port group has been added to an existing vDS with the new VLAN/subnet configured.
  • /VSAN/IgnoreClusterMemberListupdates is set to the default value of 0. If this setting is not 0, vCenter will not be able to push out any updates to the unicastagent list on the ESXi hosts resulting in a production outage.
  • Run the following command on all hosts in the cluster to check the current setting:
    esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListupdates
  • If the value is 1 please set it back to the default value of 0 via the following command:
    esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates

 

Steps required

  • First, we need to list the VMKs currently configured.
    • In this sample configuration, we have 4 hosts with IPs ending in -- .81  .82  .83  and  .84

  • SSH to each host in your vSAN Cluster, login as root and type the following command:
    esxcli network ip interface ipv4 get
     
    You'll see output similar to below:
    Name  IPv4 Address   IPv4 Netmask   IPv4 Broadcast  Address Type  Gateway      DHCP DNS
    ----  -------------  -------------  --------------  ------------  -----------  --------
    vmk0  192.168.0.81    255.255.255.0  192.168.0.255    STATIC        192.168.0.1   false
    vmk1  192.168.1.81    255.255.255.0  192.168.1.255    STATIC        0.0.0.0       false    <<----VLAN10
    vmk2  192.168.2.81    255.255.255.0  192.168.2.255    STATIC        0.0.0.0       false    <<----VLAN20

  • Next, we need to know the VMK currently being used by vSAN:
    esxcli vsan network list
Interface
   VmkNic Name: vmk2     <<----This is the VMK interface used for VSAN
   IP Protocol: IP
   Interface UUID: ########-####-####-####-############
   Agent Group Multicast Address: 224.2.3.4
   Agent Group IPv6 Multicast Address: 2001:db8::2:3:4
   Agent Group Multicast Port: 23451
   Master Group Multicast Address: 224.1.2.3
   Master Group IPv6 Multicast Address: 2001:db8::1:2:3
   Master Group Multicast Port: 12345
   Host Unicast Channel Bound Port: 12321
   Multicast TTL: 5
   Traffic Type: vsan
 
  • To move the vSAN Data to the new VLAN, follow these steps:
    • In the vSphere Web Client, create a new VMkernel Interface on each vSAN host - for this KB the new interface is: vmk3
    • Configure the IP Addresses from your new VLAN on these new VMKs for each host, being sure to select the correct vSwitch and/or Portgroup.
    • Enable vSAN on the new VMKs for each host
    • You should now see the new VMK from the CLI with the following command:
      esxcli network ip interface ipv4 get
       
      You'll see output similar to below.
       
      Name  IPv4 Address   IPv4 Netmask   IPv4 Broadcast  Address Type  Gateway      DHCP DNS
      ----  -------------  -------------  --------------  ------------  -----------  --------
      vmk0  192.168.0.81    255.255.255.0  192.168.0.255    STATIC        192.168.0.1     false
      vmk1  192.168.1.81    255.255.255.0  192.168.1.255    STATIC        0.0.0.0         false    <<----VLAN10
      vmk2  192.168.2.81    255.255.255.0  192.168.2.255    STATIC        0.0.0.0         false    <<----VLAN20
      vmk3  192.168.3.81    255.255.255.0  192.168.3.255    STATIC        0.0.0.0         false    <<----New VMK on VLAN30
  • Now check to make sure vSAN is aware that both VMKs are tagged for vSAN traffic:
    esxcli vsan network list 

  • You should now see output listing two VMKs similar to below.
    Interface
       VmkNic Name: vmk2     <<----Original vSAN VMK on Old VLAN
       IP Protocol: IP
       Interface UUID: ########-####-####-####-############
       Agent Group Multicast Address: 224.2.3.4
       Agent Group IPv6 Multicast Address: 2001:db8::2:3:4
       Agent Group Multicast Port: 23451
       Master Group Multicast Address: 224.1.2.3
       Master Group IPv6 Multicast Address: 2001:db8::1:2:3
       Master Group Multicast Port: 12345
       Host Unicast Channel Bound Port: 12321
       Multicast TTL: 5
       Traffic Type: vsan  <<----Tagged for vSAN traffic
     
    Interface
       VmkNic Name: vmk3   <<----New vSAN VMK configured on the new VLAN
       IP Protocol: IP
       Interface UUID: ########-####-####-####-############
       Agent Group Multicast Address: 224.2.3.4
       Agent Group IPv6 Multicast Address: 2001:db8::2:3:4
       Agent Group Multicast Port: 23451
       Master Group Multicast Address: 224.1.2.3
       Master Group IPv6 Multicast Address: 2001:db8::1:2:3
       Master Group Multicast Port: 12345
       Host Unicast Channel Bound Port: 12321
       Multicast TTL: 5
       Traffic Type: vsan   <<----Tagged for vSAN traffic

  • Next, we will test connectivity between all hosts on the new VMK interface (in this example it's vmk3).
    This command will use the new VMK interface (vmk3) to ping the IP addresses of vmk3 on the other host.
    • If using MTU 1500:
      vmkping -I vmk3 -s 1472 -d 192.168.3.82
      vmkping -I vmk3 -s 1472 -d 192.168.3.83
      vmkping -I vmk3 -s 1472 -d 192.168.3.84
    • If  using  MTU 9000(Jumbo frames):
      vmkping -I vmk3 -s 8972 -d 192.168.3.82
      vmkping -I vmk3 -s 8972 -d 192.168.3.83
      vmkping -I vmk3 -s 8972 -d 192.168.3.84
       
      Output will look similar to this on each ping:
      64 bytes from 192.168.3.82: icmp_seq=0 ttl=64 time=0.933 ms
      64 bytes from 192.168.3.82: icmp_seq=1 ttl=64 time=0.908 ms
      64 bytes from 192.168.3.82: icmp_seq=2 ttl=64 time=0.735 ms
       
      --- ping statistics ---
      3 packets transmitted, 3 packets received, +1 duplicates, 0% packet loss
      round-trip min/avg/max = 0.735/1.204/1.035 ms

  • When all hosts can ping each other on the new VMK interface, starting with one host:
    • Go to the vSphere Web Client
    • Highlight the host in the Inventory Pane
    • Click the Configure Tab
    • Under "Networking" highlight VMkernel Adapters
    • Choose the old vmk (vmk2 in this sample) and click Edit (pencil icon)
    • Remove the check mark next to vSAN - this disables vSAN traffic on the original VMK

  • Verify all VMs are in an accessible state - this is precautionary, the expectation is that no VMs will be affected.
    • If VMs start becoming non-compliant, shutting down, inaccessible, etc.
    • Edit the old VMK and check the vSAN box
    • Everything should recover immediately after turning vSAN back on for the original VMK interface.

  • When the original VMK on all hosts has had vSAN disabled (check mark removed in edit mode) and all VMs/hosts are still compliant/operating as expected....
    • Delete the old VMK Interface**

Additional Information

  • If the steps in this KB Article are followed precisely, there is no risk to network disruption or data availability.
  • The change is designed to be seamless, without impact to running Virtual Machines.
  • As a precaution however, this KB advises disabling vSAN on the original VMK interface one host at a time, then briefly monitoring for evidence of VMs becoming non-compliant, shutting down, or becoming Inaccessible, etc. - again this should not happen.
  • For a more cautious approach, when it's time to disable vSAN on the original VMK interfaces, you can place the host in maintenance mode, disable the original VMK, re-test ping connectivity to all hosts on the new VLAN, then exit maintenance mode and repeat for the next host.

Powered by Wolken Software