vCenter constantly tries to connect to an unreachable IP

Products

• VMware vCenter Server
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x
• VMware vSphere ESXi

Issue/Introduction

The vSphere vCenter, constantly tries to connect to an unknown IP address, which is unreachable and is not used by the infrastructure.
We can see this traffic from the firewall logs (Firewall here is an external network firewall):

XXXX-XX-XX XX:XX:XX.XXXXXX port11 in 802.1Q vlan#### P0 arp who-has ###.##.##.### tell ###.##.##.###
XXXX-XX-XX XX:XX:XX.XXXXXX Bond2 in 802.1Q vlan#### P0 arp who-has ###.##.##.### tell ###.##.##.###

There is no 3rd party plugins or extensions installed on vCenter.

Environment

vSphere 7.0
vSphere 8.0

Cause

On the vSphere vCenter appliance:

/var/log/vmware/vmware-sps/sps.log 

####-##-##T##:##:##.###Z [pool-27-thread-1] ERROR opId=sps-Main-605126-701 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: No route to host (Host unreachable) occured as provider: https://###.##.##.###:9083/version.xml is offline
####-##-##T##:##:##.###Z [pool-27-thread-1] ERROR opId=sps-Main-605126-701 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: No route to host (Host unreachable) occured as provider: https://###.##.##.###:9083/version.xml is offline
####-##-##T##:##:##.###Z [pool-27-thread-4] ERROR opId=sps-Main-605126-701 com.vmware.vim.sms.provider.vasa.event.EventDispatcher - Error: org.apache.axis2.AxisFault: No route to host (Host unreachable) occurred as provider: https://###.##.##.###:9083/version.xml is offline
####-##-##T##:##:##.###Z [pool-27-thread-1] ERROR opId=sps-Main-605126-701 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: No route to host (Host unreachable) occured as provider: https://###.##.##.###:9083/version.xml is offline
####-##-##T##:##:##.###Z [pool-27-thread-5] ERROR opId=sps-Main-605126-701 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: No route to host (Host unreachable) occured as provider: https://###.##.##.###:9083/version.xml is offline

 

There is still a component (storage provider) configured in vCenter that is trying to reach the relevant IP.

For example in the vSphere Client -> vCenter -> configure -> Storage Providers, there is a storage provider that is in status "offline" and "Certificate expired". 

 

When the ESX hosts were checked in the vSphere environment there was no output from the following commands:

esxcli storage vvol vasaprovider list
esxcli storage vvol storagecontainer list
esxcli storage vvol protocolendpoint list

Resolution

The VASA Provider cert is expired, which if in use would affect the manageability of the Environment.

The VASA Provider is not in use from vvol perspective, and it its not registered on the hosts.

Please confirm with VASA provider vendor and remove the VASA providers registration from VC.