Duplicate IP troubleshooting methods using arping, powercli and vSphere.

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This article covers some duplicate IP troubleshooting methods using arping, powercli, and vSphere.

Environment

Bosh deployed on Vsphere

Cause

• Appliance deployed with IP address from the pool for Bosh deployments
• IaaS outage has lead to re-creating already existing Bosh instances. 
• Any other reason which may lead to presence of duplicate IP address in the environment. 

Resolution

• To check whole /24 subnet using arping for duplicate IP addresses:

1. SSH to any instance in the foundation (diego_cell, mysql VM, bosh director, etc.)
2.  sudo - i
3.  cat > arpingtest.sh
4.  paste the content of the script in
5.  Ctrl +C
6.  cat arpingtest.sh and verify content is correct
7.  chmod 777 arpingtest.sh
8.  ./arpingtest.sh > arping.txt
9.  collect the output and investigate 

Script:

#!/bin/bash
for ip in `seq 1 254`
do
arping -I eth0 -c 2 -w 1 -D 10.196.170.$ip
done

Note: you need to replace the first 3 decimal values of the IP address with the correct value for the your environment.

To get how many appliances use each IP (expected is 1 and anything bigger is a problem):     

grep reply arping.txt |awk {'print $4'}| uniq -c
      1 10.193.72.1
      1 10.193.72.6 <--- This is expected. It shows that one appliance owns this IP
      2 10.193.72.21 <--- This is a problem. It shows that 2 appliances own the same IP. 

To get the IPs with the corresponding MAC address of the owners. Here we can see that IP 10.193.72.21 has two owners with the following MAC addresses: [00:50:56:AC:B0:91]  and [00:50:56:AC:B8:75]

grep reply arping.txt |awk {'print $4, $5'}

10.193.72.1 [02:E0:52:72:E6:48]
10.193.72.6 [00:50:56:AC:E8:44]
10.193.72.21 [00:50:56:AC:B0:91] <--- Appliance which owns the same IP as the next one.
10.193.72.21 [00:50:56:AC:B8:75} <--- Appliance which owns the same IP as the previous one. 

Get the MAC address of the correct VM and try to find who owns the offending MAC address.:

This can be done with PowerCLI as well: 

Get-VM | `
  Get-NetworkAdapter | `
  Where-Object {$_.MacAddress -eq "00:50:56:AC:B8:75"} | `
  Format-List -Property *

Note: PowerCLI is usually found as a module for PowerShell. You need VCenter credentials.
 

Note: that PowerCLI will help only if:

1: the mac starts with 00:50:56 - vSphere VM MAC address
2: The offending VM is part of the same VCenter (if multiple VCenters run the powercli script for each one of them)
3: the vCenter user used has permissions over all VMs in the environment.If the mac starts with something else, it can be checked in tool like: https://macvendors.com/ . This tool can show who was the manufacturer of the appliance. This will make it easier to find what device might own the duplicate IP this way.
 

Note: Arping is layer 2 tool and works only if ran from device on the same network and VLAN/VXLAN as the suspected IP address. This exact script will work only for /24 or smaller subnets.

You can also try to search for the IP address in vSphere inventory search box, however this is not always reliable and might not pick up the offending appliance.