Troubleshooting Intermittent gaps with Ceilometer data gathering.

Products

• VMware Integrated OpenStack

Issue/Introduction

The purpose of this article is to troubleshoot and determine if there are any gaps in the data that is gathered by Ceilometer in VMware Integrated Openstack.

 

Environment

VIO 7.x

Cause

• Customers may use third-party monitoring tools such as Zabbix to fetch data from Ceilometer. The monitoring tool may have challenges in obtaining data from the Ceilometer or may be missing data for a specific interval. 

Resolution

1. Check using "osctl get pods" to determine if any Ceilometer or Gnocchi pods are in the init or evicted state or have been recreated recently.
2. Check if there is enough space in Maria database "osctl exec -it mariadb-server-0 -- df -h". 

3. Verify if the archive polices and metrics

   •  openstack metric archive-policy list

   •   open stack metric list

   •   openstack metric archive-policy show <archive-policy-name>.
     
     The output of `openstack metric list` will also include `archive_policy/name` field, allowing you to determine the relevant archive policy.
     
     

4. Check the data retention time for the metrics for a period of one hour using the below command
   
   
   • openstack metric measures show <metric_id> --start "$(date -u -d '1 hour ago' '+%Y-%m-%dT%H:%M:%SZ')" --stop "$(ddate -u '+%Y-%m-%dT%H:%M:%SZ')"
     
     +---------------------------+-------------+----------------+
     | timestamp                 | granularity |          value |
     +---------------------------+-------------+----------------+
     | 2025-02-26T04:20:00+00:00 |       300.0 | 107374182400.0 |
     | 2025-02-26T04:35:00+00:00 |       300.0 | 107374182400.0 |
     | 2025-02-26T04:40:00+00:00 |       300.0 | 107374182400.0 |
     | 2025-02-26T04:55:00+00:00 |       300.0 | 107374182400.0 |
     | 2025-02-26T05:00:00+00:00 |       300.0 | 107374182400.0 |
     | 2025-02-26T05:15:00+00:00 |       300.0 | 107374182400.0 |  
     +---------------------------+-------------+----------------+ 
     
     
   • The metric_id can be any one of the metrics from `openstack metric list`.
   • The 'granularity' value indicates the time window (in seconds) over which that data point was aggregated.  A granularity of 300.0 means each value represents data over a 5‑minute interval (aggregated every 5 min).
   • Additionally, the 'timestamps' column shown by the command indicates when the metric values were recorded, and the overall span of these timestamps reflects the metric’s retention period.
   • If any inconsistency is observed, then consider disabling and enabling Ceilometer from the Horizon UI.
   • If there is no inconsistency issue observed, we can determine that Ceilometer is functioning as expected. This is most likely an issue with the third-party monitoring tool that fetches data from Ceilometer.
   • Monitoring tools have multiple ways to fetch data from OpenStack metrics—it can directly call OpenStack APIs, use third-party plugins, go through intermediate services(script) or use an intermediate proxy and needs to be investigated by the monitoring tool vendor.