Services will stop and NTPSEC service is unhealthy in Platform/Collector when NTP is not in sync for VCF Operations for Networks
Article ID: 340847
Updated On:
Products
Issue/Introduction
- The services in VCF Operations for Networks will stop and may fail to restart if not synchronized with NTP.
- When you run the
check-service-healthcommand, you may encounter an output similar to this when the system is not in sync with the NTP.
Refer to output as below:
ubuntu@aria-networks-platform:~$ ./check-service-health.sh -p -d
ElasticSearch is running and healthy.
ElasticSearch statistics:
Uptime:19:14:30
Problem: HMaster javaservice is not running.
Problem: HRegionServer javaservice is not running.
Problem: QuorumPeerMain javaservice is not running.
Problem: ResourceManager javaservice is not running.
Problem: NodeManager javaservice is not running.
Problem: Launcher javaservice is not running.
Problem: VIPService javaservice is not running.
Problem: DatabusGateway javaservice is not running.
Problem: FlinkContainer javaservice is not running.
Problem: SaasListener javaservice is not running.
Problem: Kafka javaservice is not running.
Problem: Restapilayer javaservice is not running.
Problem: TSDB javaservice is not running.
Problem: Nginx is not running.
ExpressJSApp is running
Uptime:19:14:35
NTPSEC is running but not healthy.
IMPACT: It may affect the proper working of other services.
ACTION: Restore the service using [ntp] CLI.
Uptime:06:58
FoundationDB is running and healthy.
FoundationDB statistics:
Uptime:17:55:16
17:55:16
ubuntu@aria-networks-platform:~$
During the initial setup of VCF Operations for Networks, after providing the NTP server IP address, you nay encounter: "NTP not synced"
You may receive errors in the GUI like the following which do not disappear after 6 or more hours:
- Operations for Networks System (Collector_##.##.###.###)
One or more essential services are not healthy.
Resolution: NTPSEC service is not healthy on aria-networks-collector (##.##.###.###). If the situation persists for more than 6 hours, contact VMware customer support. - Operations for Networks System (platform#)
One or more essential services are not healthy.
Resolution: NTPSEC service is not healthy on platform#(##.##.###.###). If the situation persists for more than 6 hours, contact VMware customer support.
NOTE: VCF Operations for Networks was formerly named Aria Operations for Networks (AON), and prior to that was named vRealize Network Insight (vRNI).
Cause
In VCF Operations for Networks, the deployment setup must be synchronized with the configured NTP.
If the NTP is out of sync, the services will stop and cannot be restarted within VCF Operations for Networks.
Resolution
Ensure that the connectivity to the NTP server is functional. The NTP server communicates over UDP Port 123.
1. Verify the NTP configuration by running the following command on the VCF Operations for Networks appliance (platform/collector) while logged in via SSH as "consoleuser", using the "cli" commands:
(cli) ntp-show
You see output similar to:
(cli) ntp show
Configured NTP server(s):
server ntp.###.net iburst
ntp.###.net
Status: NOT-IN-SYNC
2. If any changes are required in the NTP configuration, follow these steps:
Log in to the Platform/Collector as consoleuser.
Run the following command to configure the NTP server:
(cli) ntp set --ip-fqdn <IP addresses or FQDN details of the NTP server(s)>
If your NTP server supports secure NTP, use the following command:
(cli) ntp set --ip-fqdn <IP addresses or FQDN details of the NTP server(s)> --secure
To synchronize the NTP, run the following command:
(cli) ntp sync
Troubleshooting NTP Sync Issues (If the issue persists):
Log in to the VCF Operations for Networks (Platform or Collector, depending on which node is showing the issue) as support user.
Run the following command:
support@platform1:~$ ntpq -np
The output should be similar to one of the following scenarios:
Scenario 1: NTP IN SYNC [Working]
This indicates that NTP packets are being exchanged between the client and the server, and the synchronization is functioning properly.
remote refid st t when poll reach delay offset jitter
==============================================================================
*10.##.###.# 10.##.###.# u 25 64 377 34.287 -6.555 9.710
support@vrni-platform-release:~$ sudo tcpdump -p -i eth0 udp port 123
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:50:55.957551 IP ##.###.##.134.ntp > ########.com.ntp: NTPv4, Client, length 48
09:50:55.993224 IP ########.com.ntp>##.###.##.134.ntp: NTPv3, Server, length 48
09:50:57.957546 IP ##.###.##.134.ntp > ########.com.ntp: NTPv4, Client, length 48
09:50:57.992278 IP ########.com.ntp > ##.###.##.134.ntp: NTPv3, Server, length 48
Scenario 2: NTP NOT IN SYNC [Not Working]
When you run the command ntpq -np, you may see an output like this:
remote refid st t when poll reach delay offset jitter
==============================================================================
216.##.##.## .INIT. 16 u - 64 0 0.000 0.000 0.000
Upon running tcpdump, you notice no packets or responses are coming from the NTP server:
support@vrni-platform-release:~$ sudo tcpdump -p -i eth0 udp port 123
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:54:30.032549 IP 192.168.##.##.ntp > ########.com.ntp: NTPv4, Client, length 48
09:55:35.032471 IP 192.168.##.##.ntp > ########.com.ntp: NTPv4, Client, length 48
09:56:40.032483 IP 192.168.##.##.ntp >########.com.ntp: NTPv4, Client, length 48
This indicates that the client is sending NTP requests but not receiving any responses from the NTP server. Please contact your NTP team to resolve this issue.
Scenario 3:
During the initial deployment of the platform/collector, when you run the setup command, you will be prompted to enter the NTP IP address and NTP type.
Select "Yes" if your NTP server supports secure NTP, and select "No" if it does not.
If your NTP IP address is NOT secure, and you select "Yes", then NTP will typically show status as "NOT IN SYNC".
To remedy this, if you wish to start over and redeploy selecting "No", that is one option.
However, you can instead, modify the "ntp.conf" file which is contained in the "/etc" directory, using an editor such as "vi":
support@platform1:~$ vi /etc/ntp.conf
If you answered "Yes" to the setup question about whether the NTP server is a secure NTP server, the last line will contain the keyword "nts".
To change the file as if you had answered "No" to the setup question about whether the NTP server is a secure NTP server, remove the keyword "nts" and save the file.
After saving the file, logout and log back in again as consoleuser, and run the "ntp sync" command using the "cli" interface, to force synchronize the time.
(cli) ntp sync
If all of the above scenarios are validated and NTP is still not in sync, please contact Broadcom Support for further investigation. For more information, see Creating and managing Broadcom support cases.
Additional Information
Feedback
