vSAN online health is changing from 100 to 99 intermittently with the error "unable to reach vcsa.vmware.com and vcsa.vmware.com.cdn.cloudflare.net."

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• vSAN online Health is changing from 100 to 99 intermittent and when we try to ping the vcsa.vmware.com and vcsa.vmware.com.cdn.cloudflare.net, it's failing.  
  
  
• Skyline Health shows following warning

Environment

VMware vSAN 8.X

Cause

• The DNS resolution requests originate from the vCenter appliance, If the address family (ipv4 or ipv6) is left generic or unspecified while querying, both resolutions will take place even when there is no IPv6 address configured. Therefore vSAN Health is unable to reach vcsa.vmware.com and vcsa.vmware.com.cdn.cloudflare.net.
  
  
• Steps to validate the communication between vCenter to vcsa.vmware.com

root@vcsa [ ~ ]# nslookup vcsa.vmware.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   vcsa.vmware.com    canonical name = vcsa.vmware.com.cdn.cloudflare.net.
Name:   vcsa.vmware.com.cdn.cloudflare.net
Address: XXX.XXX.XXX.XXX                                      <<<< The IPv4 IPs listed will be used in workaround
Name:   vcsa.vmware.com.cdn.cloudflare.net
Address: YYY.YY.Y.YYY                      <<<< The IPv4 IPs listed will be used in workaround
Name:   vcsa.vmware.com.cdn.cloudflare.net
Address: 2##6:4700:#::##
Name:   vcsa.vmware.com.cdn.cloudflare.net
Address: #a#6:98cl:##::a5

• nslookup using ipv6 address to resolve the name vcsa.vmware.com

root@vcsa [ ~ ]# traceroute vcsa.vmware.com.cdn.cloudflare.net
traceroute to vcsa.vmware.com.cdn.cloudflare.net. (1#2.##.#.1#5), 30 hops max, 60 byte packets
 1  _gateway (###.###.#.#)  1.085 ms  1.017 ms  0.981 ms
 2  *    *    *
 3  *    *    *
 4  *    *    *
 5  *    *    *
 6  *    *    *
 7  *    *    *
 8  *    *    *
 9  *    *    *
10  *    *    *
11  1#2.##.#.1#5 (1#2.##.#.1#5)  10.871 ms  11.111 ms  10.634 ms

• Traceroute works for ipv4 address 

root@vcsa [ ~ ]# ping vcsa.vmware.com
ping: connect: Cannot assign requested address 

• Ping to name dose not work as it is trying to use ipv6 address 

root@vcsa [ ~ ]# ping 1#2.##.#.1#5 -c 16
PING 1#2.##.#.1#5 (172.66.0.165) 56(84) bytes of data.
64 bytes from 1#2.##.#.1#5: icmp_seq=1 ttl=48 time=11.0 ms
64 bytes from 1#2.##.#.1#5: icmp_seq=2 ttl=48 time=11.0 ms
64 bytes from 1#2.##.#.1#5: icmp_seq=3 ttl=48 time=11.4 ms
64 bytes from 1#2.##.#.1#5: icmp_seq=4 ttl=48 time=12.0 ms
[cont...]
--- 172.66.0.165 ping statistics ---
16 packets transmitted, 16 received, 0% packet loss, time 15020ms
rtt min/avg/max/mdev = 10.608/11.476/13.755/0.785 ms

root@vcsa [ ~ ]# ping 1#2.1##.1##.1#7 -c 4
PING 1#2.1##.1##.1#7 (1#2.1##.1##.1#7) 56(84) bytes of data.
64 bytes from 1#2.1##.1##.1#7: icmp_seq=1 ttl=48 time=9.54 ms
64 bytes from 1#2.1##.1##.1#7: icmp_seq=2 ttl=48 time=7.83 ms
64 bytes from 1#2.1##.1##.1#7: icmp_seq=3 ttl=48 time=8.25 ms
64 bytes from 1#2.1##.1##.1#7: icmp_seq=4 ttl=48 time=8.22 ms

--- 1#2.1##.1##.1#7 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 7.826/8.458/9.544/0.648 ms

• Ping works for the ipv4 address
  
  
• The Errors that can be observed in vsan-mgmt logs are as follows: 

INFO vsan-mgmt[3002679] [VsanSupportBundleHelper::parseSystemProxies opID=noOpId] VCSA proxy is disabled.
ERROR vsan-mgmt[3002679] [VsanHttpRequestWrapper::urlopen opID=noOpId] Exception while sending request : <urlopen error [Errno 99] Cannot assign requested address>

[cont...]

INFO vsan-mgmt[109080] [VsanSupportBundleHelper::parseSystemProxies opID=78364062] VCSA proxy is disabled.
INFO vsan-mgmt[109080] [VsanVcObjectHelper::wrapper opID=78364062] Finish execute heckHostNameResolutionEnabled


Resolution

This is a known issue and here are the workarounds: 

Workarounds: 

Option 1: 

• Modify /etc/hosts as below in the vCenter and disable the IPV6 address.
  
  
• Before editing the /etc/hosts file in the vCenter server appliance. 

root@vcsa [ ~ ]# cat /etc/hosts
# Begin /etc/hosts (network card version)

# VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1  vcsa.m####t.ch vcsa localhost
::1  vcsa.m####t.ch vcsa localhost ipv6-localhost ipv6-loopback
# VAMI_EDIT_END

• After editing the /etc/hosts file in the vCenter server appliance. 

root@vcsa [ ~ ]# cat /etc/hosts
# Begin /etc/hosts (network card version)

# VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1  vcsa.m####t.ch vcsa localhost
#::1  vcsa.m####t.ch vcsa localhost ipv6-localhost ipv6-loopback                 <<<<<< Line Commented
1#2.1##.1##.1#7 vcsa.vmware.com                    <<<<<< Line added (Please use IPv4 address XXX.XXX.XXX.XXX or YYY.YY.Y.YYY)
# VAMI_EDIT_END

Option 2:

Note: If Option 1 does not work then follow the below steps:

• Upgrade the vCenter to 8.0U3D and then add the filter-AAAA to the dnsmasq.conf file on the vCenter Server.
  
  
• filter-AAAA will not work for dnsmasq >= 2.87 which is available in 80U3(dnsmasq-2.90-1.ph4.x86_64.rpm ) and above.
  
  
• Before modifying /etc/dnsmasq.conf

root@vcsa [ ~ ]# cat /etc/dnsmasq.conf
listen-address=127.0.0.1
bind-interfaces
user=dnsmasq
group=dnsmasq

no-hosts
log-queries
log-facility=/var/log/vmware/dnsmasq.log
domain-needed
dns-forward-max=150
cache-size=8192
neg-ttl=3600
root@vcenter [ ~ ]# 

• After modifying the config 

root@vcsa [ ~ ]# cat /etc/dnsmasq.conf
listen-address=127.0.0.1
bind-interfaces
user=dnsmasq
group=dnsmasq

no-hosts
log-queries
log-facility=/var/log/vmware/dnsmasq.log
domain-needed
dns-forward-max=150
cache-size=8192
neg-ttl=3600
filter-AAAA                 <<<<<< Line added 

• Restart the following services on vCenter server. 

root@vcsa [ ~ ]# kill -SIGUSR1 `pgrep dnsmasq`
root@vcsa [ ~ ]# systemctl daemon-reload
root@vcsa [ ~ ]# systemctl restart dnsmasq