The capc-consul service fails to run with error messages in Performance Center
Article ID: 185320
Updated On:
Products
Issue/Introduction
After upgrading the capc-consul service on DX NetOps Performance Management Performance Center doesn't run.
For an unknown reason, the following error messages are getting sporadically registered at capc:
Matched Text: Feb 23 17:56:44 ##### consul: 2020/02/23 17:56:44 [ERR] yamux: keepalive failed: session shutdown
Matched Text: Feb 23 17:56:44 ##### consul: 2020/02/23 17:56:44 [ERR] consul.rpc: multiplex conn accept failed: read tcp x.x.x.x:8700->x.x.x.x:52218: read: connection reset by peer from=x.x.x.x:52218
Matched Text: Feb 21 17:15:01 ##### consul: 2020/02/21 17:15:01 [ERR] yamux: Failed to write header: write tcp x.x.x.x:42100->x.x.x.x:8700: use of closed network connection
NOTE: x.x.x.x – IP address of CAPC
Messages like these could also be seen in the Performance Center systems /var/log/messages log files.
Mar 15 16:29:24 ##### consul: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Mar 15 16:29:24 ##### consul: bootstrap = true: do not enable unless necessary
Mar 15 16:29:24 ##### consul: ==> Starting Consul agent...
Mar 15 16:29:24 ##### consul: Version: 'v1.7.2'
Mar 15 16:29:24 ##### consul: Node ID: 'ec9d62e5-0835-1ef1-a097-5458b1117ad5'
Mar 15 16:29:24 ##### consul: Node name: '#####'
Mar 15 16:29:24 ##### consul: Datacenter: 'capc' (Segment: '<all>')
Mar 15 16:29:24 ##### consul: Server: true (Bootstrap: true)
Mar 15 16:29:24 ##### consul: Client Addr: [x.x.x.x] (HTTP: 8900, HTTPS: -1, gRPC: -1, DNS: 9000)
Mar 15 16:29:24 ##### consul: Cluster Addr: x.x.x.x (LAN: 8701, WAN: 8702)
Mar 15 16:29:24 ##### consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
Mar 15 16:29:24 ##### consul: ==> Log data will now stream in as it occurs:
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.538+0100 [INFO] agent.server.raft: restored from snapshot: id=9-671753-1582881210513
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.666+0100 [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:x.x.x.x:8700 Address:x.x.x.x:8700}]"
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.667+0100 [INFO] agent.server.raft: entering follower state: follower="Node at x.x.x.x:8700 [Follower]" leader=
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.690+0100 [INFO] agent.server.serf.wan: serf: EventMemberJoin: #####.capc x.x.x.x
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.690+0100 [WARN] agent.server.serf.wan: serf: Failed to re-join any previously known node
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.711+0100 [INFO] agent.server.serf.lan: serf: EventMemberJoin: ##### x.x.x.x
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.713+0100 [WARN] agent.server.serf.lan: serf: Failed to re-join any previously known node
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.713+0100 [INFO] agent.server: Adding LAN server: server="##### (Addr: tcp/x.x.x.x:8700) (DC: capc)"
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.713+0100 [INFO] agent.server: Handled event for server in area: event=member-join server=#####.capc area=wan
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.713+0100 [INFO] agent: Started DNS server: address=x.x.x.x:9000 network=udp
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.714+0100 [INFO] agent: Started DNS server: address=x.x.x.x:9000 network=tcp
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.714+0100 [INFO] agent: Started HTTP server: address=x.x.x.x:8900 network=tcp
Mar 15 16:29:24 ##### consul: 2021-03-15T16:29:24.714+0100 [INFO] agent: started state syncer
Mar 15 16:29:24 ##### consul: ==> Consul agent running!
Mar 15 16:29:31 ##### consul: 2021-03-15T16:29:31.792+0100 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Mar 15 16:29:34 ##### consul: 2021-03-15T16:29:34.186+0100 [WARN] agent.server.raft: not part of stable configuration, aborting election
Mar 15 16:29:42 ##### systemd: Reloading.
Mar 15 16:29:42 ##### systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:29:42 ##### systemd: Reloading.
Mar 15 16:29:42 ##### systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:29:53 ##### consul: 2021-03-15T16:29:53.088+0100 [ERROR] agent: Failed to check for updates: error="Get https://www.<host name>.<Your domain>: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 15 16:29:54 ##### consul: 2021-03-15T16:29:54.112+0100 [ERROR] agent: Coordinate update error: error="No cluster leader"
Mar 15 16:30:00 ##### consul: 2021-03-15T16:30:00.492+0100 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Mar 15 16:30:06 ##### systemd: Reloading.
Mar 15 16:30:06 ##### systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:06 ##### systemd: Reloading.
Mar 15 16:30:06 ##### systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:20 ##### consul: 2021-03-15T16:30:20.405+0100 [ERROR] agent: Coordinate update error: error="No cluster leader"
Mar 15 16:30:22 ##### systemd: Reloading.
Mar 15 16:30:22 ##### systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:23 ##### systemd: Reloading.
Mar 15 16:30:23 ##### systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:36 ##### consul: 2021-03-15T16:30:36.506+0100 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Mar 15 16:30:39 ##### systemd: Reloading.
Mar 15 16:30:39 ##### systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:39 ##### systemd: Reloading.
Mar 15 16:30:39 ##### systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:46 ##### consul: 2021-03-15T16:30:46.381+0100 [ERROR] agent: Coordinate update error: error="No cluster leader"
Environment
All supported DX NetOps Performance Management releases
Cause
capc-consul service is not being used as of now in the DX NetOps Performance Management Performance Center servers.
Resolution
Currently we are not using the capc-consul service on Performance Center. It will be used in the future for a DX NetOps Performance Management Performance Center Fault Tolerance solution.
At this time please ignore those messages. The recommendation is to disable the capc-consul service on the Performance Center server to prevent it from being used at this time.
First ensure it's stopped. It may be seen as active but the service itself may not be running. To do this run:
systemctl stop capc-consul
Now run the following to disable the service from being run.
systemctl disable capc-consul
Additional Information
If running older RH 6.x OS use the following to disable the service.
- Disable the capc-consul service:
- chkconfig capc-consul off
- View service state:
- chkconfig capc-consul --list
From the Jobs and Services section of the "Common administrative commands in Red Hat Enterprise Linux 5, 6, 7, and 8" page for RedHat.
Feedback
