Usage of Timestamp Fields from NetOps Events (cycleTimestampMS vs readTimestampMS)

Products

• Network Observability
• CA Performance Management

Issue/Introduction

Using the timestamp fields from events exported from NetOps to Kafka, we are exploring the possibility of using cycleTimestampMS as the primary time field (instead of @timestamp / readTimestampMS) in ElasticSearch to better reflect the actual event time.

However, during analysis, some inconsistencies between the timestamp fields are observed:

1. In several samples, the value of cycleTimestampMS is significantly ahead of the current timestamp, which may result in events being indexed as “future” data if used as the primary time field. What is the exact definition and source of cycleTimestampMS & readTimestampMS?
   
   
2. The value of readTimestampMS (currently mapped to @timestamp in ElasticSearch) can differ by several hours compared to cycleTimestampMS.
   
   
3. If cycleTimestampMS is the more accurate, can it be used in ElasticSearch instead of readTimestampMS?

Environment

DX NetOps CAPM all currently supported releases

Resolution

We provide views to show the data, which would show a data point at next 5 min mark depending when it was polled and loaded during the poll cycle. 

The readTimestampMS is the actual time data was polled from the device, while cycleTimestampMS is a normalised value representing the end of a collection cycle. Future-dated cycleTimestampMS is expected behaviour due to data loading thresholds (400MB or 5-minute windows).

We don't record the readTimestampMS anywhere except in memory logging in dcdebug. The DB only has cycleTimestampMS

So yes, we can confirm that cycleTimestampMS can be used as the primary time field for @timestamp in Elasticsearch.

Also, to ensure minimal differences in times recorded, ALL servers should be in time-sync, as per:

TechDocs : DX NetOps 25.4 : PM Installation Requirements - Common Considerations

• NetOps Portal
  
  NetOps Portal, the data repository, the data aggregator, and the data collectors require time synchronization using the Network Time Protocol (NTP) networking protocol. If it is not running, start the NTP daemon. All machines must use the same NTP server.

They don't need to be in the same timezone, BUT they need to be less than 1-2 secs apart. If you update the time on a machine for DA or DC or Portal, restart services