The NetOps Portal consul service fails to run with error messages in the NetOps Portal
Article ID: 185320
Updated On:
Products
Issue/Introduction
After upgrading the NetOps Portal consul service on DX NetOps Performance Management NetOps Portal doesn't run.
For an unknown reason, the following error messages are getting sporadically registered at the NetOps Portal:
Matched Text: Feb 23 17:56:44 ##### consul: 2020/02/23 17:56:44 [ERR] yamux: keepalive failed: session shutdown
Matched Text: Feb 23 17:56:44 ##### consul: 2020/02/23 17:56:44 [ERR] consul.rpc: multiplex conn accept failed: read tcp x.x.x.x:8700->x.x.x.x:52218: read: connection reset by peer from=x.x.x.x:52218
Matched Text: Feb 21 17:15:01 ##### consul: 2020/02/21 17:15:01 [ERR] yamux: Failed to write header: write tcp x.x.x.x:42100->x.x.x.x:8700: use of closed network connection
NOTE: x.x.x.x – IP address of NetOps Portal
Messages like these could also be seen in the NetOps Portal systems /var/log/messages log files.
Mar 15 16:29:24 ##### consul: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Mar 15 16:29:24 ##### consul: bootstrap = true: do not enable unless necessary
Mar 15 16:29:24 ##### consul: ==> Starting Consul agent...
Mar 15 16:29:24 ##### consul: Version: 'v1.7.2'
Mar 15 16:29:24 ##### consul: Node ID: 'ec9d62e5-0835-1ef1-a097-5458b1117ad5'
Mar 15 16:29:24 ##### consul: Node name: '#####'
Mar 15 16:29:24 ##### consul: Datacenter: 'netops portal' (Segment: '<all>')
Mar 15 16:29:24 ##### consul: Server: true (Bootstrap: true)
Mar 15 16:29:24 ##### consul: Client Addr: [x.x.x.x] (HTTP: 8900, HTTPS: -1, gRPC: -1, DNS: 9000)
Mar 15 16:29:24 ##### consul: Cluster Addr: x.x.x.x (LAN: 8701, WAN: 8702)
Mar 15 16:29:24 ##### consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
Mar 15 16:29:24 ##### consul: ==> Log data will now stream in as it occurs:
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.538+0100 [INFO] agent.server.raft: restored from snapshot: id=9-671753-1582881210513
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.666+0100 [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:x.x.x.x:8700 Address:x.x.x.x:8700}]"
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.667+0100 [INFO] agent.server.raft: entering follower state: follower="Node at x.x.x.x:8700 [Follower]" leader=
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.690+0100 [INFO] agent.server.serf.wan: serf: EventMemberJoin: #####.netops portal x.x.x.x
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.690+0100 [WARN] agent.server.serf.wan: serf: Failed to re-join any previously known node
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.711+0100 [INFO] agent.server.serf.lan: serf: EventMemberJoin: ##### x.x.x.x
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.713+0100 [WARN] agent.server.serf.lan: serf: Failed to re-join any previously known node
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.713+0100 [INFO] agent.server: Adding LAN server: server="##### (Addr: tcp/x.x.x.x:8700) (DC: NetOps Portal)"
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.713+0100 [INFO] agent.server: Handled event for server in area: event=member-join server=#####.NetOps Portal area=wan
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.713+0100 [INFO] agent: Started DNS server: address=x.x.x.x:9000 network=udp
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.714+0100 [INFO] agent: Started DNS server: address=x.x.x.x:9000 network=tcp
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.714+0100 [INFO] agent: Started HTTP server: address=x.x.x.x:8900 network=tcp
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.714+0100 [INFO] agent: started state syncer
Mar 15 16:29:24 ##### consul: ==> Consul agent running!
Mar 15 16:29:31 ##### consul: 2021-03-15T16:29:31.792+0100 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Mar 15 16:29:34 ##### consul: 2021-03-15T16:29:34.186+0100 [WARN] agent.server.raft: not part of stable configuration, aborting election
Mar 15 16:29:42 ##### systemd: Reloading.
Mar 15 16:29:42 ##### systemd: Configuration file /etc/systemd/system/consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:29:42 ##### systemd: Reloading.
Mar 15 16:29:42 ##### systemd: Configuration file /etc/systemd/system/consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:29:53 ##### consul: 2021-03-15T16:29:53.088+0100 [ERROR] agent: Failed to check for updates: error="Get https://www.<host name>.<Your domain>: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 15 16:29:54 ##### consul: 2021-03-15T16:29:54.112+0100 [ERROR] agent: Coordinate update error: error="No cluster leader"
Mar 15 16:30:00 ##### consul: 2021-03-15T16:30:00.492+0100 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Mar 15 16:30:06 ##### systemd: Reloading.
Mar 15 16:30:06 ##### systemd: Configuration file /etc/systemd/system/consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:06 ##### systemd: Reloading.
Mar 15 16:30:06 ##### systemd: Configuration file /etc/systemd/system/consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:20 ##### consul: 2021-03-15T16:30:20.405+0100 [ERROR] agent: Coordinate update error: error="No cluster leader"
Mar 15 16:30:22 ##### systemd: Reloading.
Mar 15 16:30:22 ##### systemd: Configuration file /etc/systemd/system/consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:23 ##### systemd: Reloading.
Mar 15 16:30:23 ##### systemd: Configuration file /etc/systemd/system/consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:36 ##### consul: 2021-03-15T16:30:36.506+0100 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Mar 15 16:30:39 ##### systemd: Reloading.
Mar 15 16:30:39 ##### systemd: Configuration file /etc/systemd/system/consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:39 ##### systemd: Reloading.
Mar 15 16:30:39 ##### systemd: Configuration file /etc/systemd/system/consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:46 ##### consul: 2021-03-15T16:30:46.381+0100 [ERROR] agent: Coordinate update error: error="No cluster leader"
Environment
All supported DX NetOps Performance Management releases
Cause
NetOps consul service is not being used as of now in the DX NetOps Performance Management NetOps Portal servers.
Resolution
Currently we are not using the consul service on the NetOps Portal. It will be used in the future for a DX NetOps Performance Management NetOps Portal Fault Tolerance solution.
At this time please ignore those messages. The recommendation is to disable the consul service on the NetOps Portal server to prevent it from being used at this time.
First ensure it's stopped. It may be seen as active but the service itself may not be running. To do this run:
systemctl stop consul
Now run the following to disable the service from being run.
systemctl disable consul
Additional Information
If running older RH 6.x OS use the following to disable the service.
- Disable the consul service:
- chkconfig consul off
- View service state:
- chkconfig consul --list
From the Jobs and Services section of the "Common administrative commands in Red Hat Enterprise Linux 5, 6, 7, and 8" page for RedHat.
Feedback
