APM - Metrics delayed in metric view and connectiosn status not updated

Products

• DX Application Performance Management

Issue/Introduction

Metrics are delayed in metric view and connection status is not updated for agents

Environment

Release : 20.2

Component :

Cause

inspection of the metadata logs from  {NFSSHARE}/apmservices/metadata-xxx/logs showed the following messages

2022-01-07 11:41:52.223  INFO 1 --- [metadata.setState-0] c.c.a.m.db.meta.MetadataManagerImpl      : Data in the past, we need to recreate history. GlobalMetricIdWithTime [tenantId=10, sourceId=15, type=1, attributeId=102170, firstSeen=1641555000, lastSeen=1641555900, reported=1641555663]
2022-01-07 11:42:08.368  INFO 1 --- [metadata.setState-3] c.c.a.m.db.meta.MetadataManagerImpl      : Data in the past, we need to recreate history. GlobalMetricIdWithTime [tenantId=10, sourceId=15, type=1, attributeId=35975, firstSeen=1641555000, lastSeen=1641555900, reported=1641555669]

checking the system time on the cluster nodes showed that they were not in synch - up to 5 minutes apart 

 

 

Resolution

Problem was resolved once cluster node clocks were synched (NTP was implemented). 

It is stipulated that there should be no time skew between nodes.
 
https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/21-3/installing/Installation-Scenarios-2131/Install-as-a-Cluster-Administrator/Kubernetes---Pre-installation-Tasks-Cluster-Admin.html#concept.dita_8a89427a-18c1-47bb-beab-cc2cc0d3c934_SetupNTP