Cloud Proxy is sending ICMP pings to a private Azure subnet or another IP address
search cancel

Cloud Proxy is sending ICMP pings to a private Azure subnet or another IP address

book

Article ID: 396585

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • You may see from firewall logs or other network monitoring that an IP address belonging to an Aria Operations cloud proxy or cluster node is sending ICMP pings to a private Azure subnet or some other endpoint IP address in your environment unexpectedly
  • Searching in Aria Operations or in vCenter does not show an object with that IP address
  • From logs you are unable to find the cause of the ICMP pings
  • A tcpdump result from the node sending the pings is showing ICMP echo requests out every 2 seconds

NOTE: tcpdump is not installed natively on any Aria Operations node due to product security requirements. If you need to troubleshoot with tcpdump, please open a case with support

Example output 

root@<ops_node_hostname> [ ~ ]# tcpdump icmp and src <ops_node_IP> and dst <target_IP>
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
23:46:15.169897 IP <ops_node_FQDN> > <target_IP_or_FQDN>: ICMP echo request, id 43793, seq 5, length 64
23:46:17.169843 IP <ops_node_FQDN> > <target_IP_or_FQDN>: ICMP echo request, id 43793, seq 7, length 64
23:46:19.169814 IP <ops_node_FQDN> > <target_IP_or_FQDN>: ICMP echo request, id 43793, seq 9, length 64
23:46:21.169932 IP <ops_node_FQDN> > <target_IP_or_FQDN>: ICMP echo request, id 43793, seq 11, length 64
23:46:23.169845 IP <ops_node_FQDN> > <target_IP_or_FQDN>: ICMP echo request, id 43793, seq 13, length 64
23:46:25.169708 IP <ops_node_FQDN> > <target_IP_or_FQDN>: ICMP echo request, id 43793, seq 15, length 64

Environment

Aria Operations 8.x

Cause

  • Ping Monitoring has been enabled for a VM with a network device that has been assigned with the IP address receiving the ICMP pings.
    • The metric on the VM that has this IP address would be Properties > Network > 4000 / 4001 / -1 > IP Address
    • In vCenter a network ID of 'net:-1' refers to an IP address defined in the guest but not associated with any vnic. This is why you may not find the IP address in vCenter when searching directly.
  • The built-in ping mechanism (fping) will try and ping all IP addresses that have been collected for this object, including those not assigned to a vnic
  • The pings will come from whatever vCenter adapter instance the VM is associated to and could be a data node or cloud proxy, but the data for the IP address is stored in the database on one or more data nodes
  • If you are not able to find the object in the Aria Operations UI manually, you can run a database query
  • To run the database query:
    • Login to an SSH session as root to each of the data nodes in the Aria Operations cluster.
    • In the case of a 3-node HA enabled cluster, the property in the database can be on any 2 of the nodes so we have to check them all.
    • Copy and paste the command below, substituting IP_ADDRESS with the endpoint IP address you are looking for (leave the single quotes)

su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -p 5432 -d vcopsdb -c \"SELECT i.resource_name, i.resource_kind_key, p.property_instance_name, p.property_value_string FROM hd_resource_property_history_v AS p JOIN resource_id_name AS i ON i.resource_id = p.internal_id WHERE p.property_name = 'net|ip_address' AND p.period_end_time = 9999999999999 AND p.property_value_string IN ('IP_ADDRESS') ORDER BY p.property_value_string;\""

    • If the command returns 0 rows, try the same command on the next cluster node (be sure to check the IP address is entered correctly)
    • Once you get a result, search for the object name under resource_name in Aria Operations

Resolution

  • With the object opened in Aria Operations Inventory, click the Summary tab and you should see the Ping Statistics widget on the bottom right of the page. If enabled and you don't need ping statistics metrics for this object, click the ellipses (3 dots) and select 'Deactivate Ping Monitoring
  • Another location for this setting is within Inventory Management > Object Types. Choose either Host System or Virtual Machine. Click on the individual object, select the pencil icon at the top of the page and the 'Enable Ping monitoring' is under Advanced Settings.
  • If Ping Monitoring is still needed for this object, check your network configuration in vCenter and remove or update the IP address in question.
Powered by Wolken Software