The capc-consul service fails to run with error messages in Performance Center

book

Article ID: 185320

calendar_today

Updated On:

Products

CA Performance Management for OpenVMS CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

After upgrade the capc-consul service on DX NetOps Performance Management Performance Center doesn't run.

For an unknown reason, the following error messages are getting sporadically registered at capc:

Matched Text: Feb 23 17:56:44 csp-pm-capc consul: 2020/02/23 17:56:44 [ERR] yamux: keepalive failed: session shutdown
Matched Text: Feb 23 17:56:44 csp-pm-capc consul: 2020/02/23 17:56:44 [ERR] consul.rpc: multiplex conn accept failed: read tcp x.x.x.x:8700->x.x.x.x:52218: read: connection reset by peer from=x.x.x.x:52218
Matched Text: Feb 21 17:15:01 lab-pm-capc consul: 2020/02/21 17:15:01 [ERR] yamux: Failed to write header: write tcp x.x.x.x:42100->x.x.x.x:8700: use of closed network connection

NOTE: x.x.x.x – IP address of CAPC

Messages like these could also be seen in the Performance Center systems /var/log/messages log files.

Mar 15 16:29:24 PC_HostName consul: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Mar 15 16:29:24 PC_HostName consul: bootstrap = true: do not enable unless necessary
Mar 15 16:29:24 PC_HostName consul: ==> Starting Consul agent...
Mar 15 16:29:24 PC_HostName consul: Version: 'v1.7.2'
Mar 15 16:29:24 PC_HostName consul: Node ID: 'ec9d62e5-0835-1ef1-a097-5458b1117ad5'
Mar 15 16:29:24 PC_HostName consul: Node name: 'PC_HostName'
Mar 15 16:29:24 PC_HostName consul: Datacenter: 'capc' (Segment: '<all>')
Mar 15 16:29:24 PC_HostName consul: Server: true (Bootstrap: true)
Mar 15 16:29:24 PC_HostName consul: Client Addr: [0.0.0.0] (HTTP: 8900, HTTPS: -1, gRPC: -1, DNS: 9000)
Mar 15 16:29:24 PC_HostName consul: Cluster Addr: <IP_Address> (LAN: 8701, WAN: 8702)
Mar 15 16:29:24 PC_HostName consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
Mar 15 16:29:24 PC_HostName consul: ==> Log data will now stream in as it occurs:
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.538+0100 [INFO]  agent.server.raft: restored from snapshot: id=9-671753-1582881210513
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.666+0100 [INFO]  agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:<IP_Address>:8700 Address:<IP_Address>:8700}]"
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.667+0100 [INFO]  agent.server.raft: entering follower state: follower="Node at <IP_Address>:8700 [Follower]" leader=
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.690+0100 [INFO]  agent.server.serf.wan: serf: EventMemberJoin: PC_HostName.capc <IP_Address>
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.690+0100 [WARN]  agent.server.serf.wan: serf: Failed to re-join any previously known node
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.711+0100 [INFO]  agent.server.serf.lan: serf: EventMemberJoin: PC_HostName <IP_Address>
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.713+0100 [WARN]  agent.server.serf.lan: serf: Failed to re-join any previously known node
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.713+0100 [INFO]  agent.server: Adding LAN server: server="PC_HostName (Addr: tcp/<IP_Address>:8700) (DC: capc)"
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.713+0100 [INFO]  agent.server: Handled event for server in area: event=member-join server=PC_HostName.capc area=wan
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.713+0100 [INFO]  agent: Started DNS server: address=0.0.0.0:9000 network=udp
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.714+0100 [INFO]  agent: Started DNS server: address=0.0.0.0:9000 network=tcp
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.714+0100 [INFO]  agent: Started HTTP server: address=0.0.0.0:8900 network=tcp
Mar 15 16:29:24 PC_HostName consul: 2021-03-15T16:29:24.714+0100 [INFO]  agent: started state syncer
Mar 15 16:29:24 PC_HostName consul: ==> Consul agent running!
Mar 15 16:29:31 PC_HostName consul: 2021-03-15T16:29:31.792+0100 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Mar 15 16:29:34 PC_HostName consul: 2021-03-15T16:29:34.186+0100 [WARN]  agent.server.raft: not part of stable configuration, aborting election
Mar 15 16:29:42 PC_HostName systemd: Reloading.
Mar 15 16:29:42 PC_HostName systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:29:42 PC_HostName systemd: Reloading.
Mar 15 16:29:42 PC_HostName systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:29:53 PC_HostName consul: 2021-03-15T16:29:53.088+0100 [ERROR] agent: Failed to check for updates: error="Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=9922c01b-16fa-d11c-6dae-886fe0d8c7dc&version=1.7.2: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 15 16:29:54 PC_HostName consul: 2021-03-15T16:29:54.112+0100 [ERROR] agent: Coordinate update error: error="No cluster leader"
Mar 15 16:30:00 PC_HostName consul: 2021-03-15T16:30:00.492+0100 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Mar 15 16:30:06 PC_HostName systemd: Reloading.
Mar 15 16:30:06 PC_HostName systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:06 PC_HostName systemd: Reloading.
Mar 15 16:30:06 PC_HostName systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:20 PC_HostName consul: 2021-03-15T16:30:20.405+0100 [ERROR] agent: Coordinate update error: error="No cluster leader"
Mar 15 16:30:22 PC_HostName systemd: Reloading.
Mar 15 16:30:22 PC_HostName systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:23 PC_HostName systemd: Reloading.
Mar 15 16:30:23 PC_HostName systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:36 PC_HostName consul: 2021-03-15T16:30:36.506+0100 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Mar 15 16:30:39 PC_HostName systemd: Reloading.
Mar 15 16:30:39 PC_HostName systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:39 PC_HostName systemd: Reloading.
Mar 15 16:30:39 PC_HostName systemd: Configuration file /etc/systemd/system/capc-consul.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Mar 15 16:30:46 PC_HostName consul: 2021-03-15T16:30:46.381+0100 [ERROR] agent: Coordinate update error: error="No cluster leader"

Cause

Consul service is not being used as of now in the DX NetOps Performance Management Performance Center servers.

Environment

All supported DX NetOps Performance Management releases

Resolution

Currently we are not using the capc-consul service on Performance Center. It will be used in the future for a DX NetOps Performance Management Performance Center Fault Tolerance solution.

At this time please ignore those messages. The recommendation is to disable the capc-consul service on the Performance Center server to prevent it from being used at this time.

First ensure it's stopped. It may be seen as active but the service itself may not be running. To do this run:

systemctl stop capc-consul

Now run the following to disable the service from being run.

systemctl disable capc-consul