Checking consul member states in CA Performance Management (CAPM) Fault Tolerant Data Aggregator systems
search cancel

Checking consul member states in CA Performance Management (CAPM) Fault Tolerant Data Aggregator systems

book

Article ID: 190207

calendar_today

Updated On: 09-17-2024

Products

CA Performance Management Network Observability

Issue/Introduction

How can the state of the Fault Tolerant Data Aggregator consul proxy be checked?

How to check the status of consul proxy?

How to check the connection status for Fault Tolerant Data Aggregators?

Environment

All supported Performance Management installations running Fault Tolerant Data Aggregators

Resolution

The commands shown below will be found in these default paths:

  1. On DA in /opt/IMDataAggregator/consul/bin
  2. On Proxy host in /opt/CA/daproxy/bin

Both commands referenced below must be run when consul proxy services are running on the host it's run on.

To check the proxy/consul membership and connection states run the 'consul members' command. In release 20.2.8 onwards, we moved to token usage for consul, so you need you need to pass -token <token> on command line. The Token is in the file:

</DASharedRepo>/acl-token.properties

So you would run the command as per the following example (Sample output from a support lab running Fault Tolerant Data Aggregators. This is from the consul proxy host.):

[root@bin]# ./consul members -token 9845e08e-0201-f027-999f-672b015bf4ce
Node                            Address            Status  Type    Build  Protocol  DC
DA1_HostName                 <IPAddress>:8301      alive   server  0.8.3  2         capm
DA2_HostName                 <IPAddress>:8301      alive   server  0.8.3  2         capm
ProxyHost                    <IPAddress>:8301      alive   server  0.8.3  2         capm


To check the state of the consul proxy and determine which DA is active run the following command.

[root@consulProxyHost bin]# ./consul operator raft list-peers -token <Token>

Notes:

  • If run before all systems are up for proxy services we'll see the following error:

    • [root@consulProxyHost bin]# ./consul operator raft list-peers
      Error getting peers: Failed to retrieve raft configuration: Unexpected response code: 500 (No cluster leader)

  • When run with everything up, one DA Active, one Inactive, the Active DA will be set as the leader and DA2 and proxy host as followers.
  • When run with only consul services running, with both DAs in Maintenance, the proxy host will show as the leader.

Sample support lab output when one DA is Active and the other is Inactive.

[root@consulProxyHost bin]# ./consul operator raft list-peers -token 9845e08e-0201-f027-999f-672b015bf4ce
Node                            ID                 Address            State     Voter  RaftProtocol
lProxyHost                      <IPAddress>:8300  <IPAddress>:8300    follower  true   2
DA1_HostName                    <IPAddress>:8300  <IPAddress>:8300    leader    true   2
DA2_HostName                    <IPAddress>:8300  <IPAddress>:8300    follower  true   2

 

Additional Information

Also make sure that both TCP and UDP ports are open for ports 8300 and 8301

If seeing warning messages due to ACL blocks due to anonymous token when checking consul status:

[root@DA2 ~/bin]$ systemctl status consul

  • consul.service - Consul Server
       Loaded: loaded (/etc/systemd/system/consul.service; enabled; vendor preset: disabled)
       Active: active (running) since Thu 2024-09-12 13:31:57 +08; 33min ago
    Main PID: 4979 (consul)
        Tasks: 13 (limit: 307057)
       Memory: 38.8M
       CGroup: /system.slice/consul.service
               └─4979 /opt/CA/IMDataAggregator/consul/bin/consul agent -bind=172.19.13.21 -disable-host-node-id -config-dir=/opt/CA/IMDataAggregator/consul/conf -log-file=/opt/CA/IMDataAggregator/consul/log/consul.log

    Sep 12 14:02:23 DA2 consul[4979]: 2024-09-12T14:02:23.105+0800 [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
    Sep 12 14:02:52 DA2 consul[4979]: 2024-09-12T14:02:52.438+0800 [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
    Sep 12 14:03:18 DA2 consul[4979]: 2024-09-12T14:03:18.954+0800 [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
    Sep 12 14:03:25 DA2 consul[4979]: 2024-09-12T14:03:25.614+0800 [WARN]  agent: Node info update blocked by ACLs: node=51c96adb-467d-8b7b-46cd-8a203472fbcf accessorID="anonymous token"
    Sep 12 14:03:36 DA2 consul[4979]: 2024-09-12T14:03:36.917+0800 [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
    Sep 12 14:04:02 DA2 consul[4979]: 2024-09-12T14:04:02.429+0800 [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
    Sep 12 14:04:23 DA2 consul[4979]: 2024-09-12T14:04:23.623+0800 [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
    Sep 12 14:04:40 DA2 consul[4979]: 2024-09-12T14:04:40.071+0800 [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
    Sep 12 14:05:01 DA2 consul[4979]: 2024-09-12T14:05:01.211+0800 [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
    Sep 12 14:05:02 DA2 consul[4979]: 2024-09-12T14:05:02.404+0800 [WARN]  agent: Node info update blocked by ACLs: node=51c96adb-467d-8b7b-46cd-8a203472fbcf accessorID="anonymous token"

Consul doesn't need the token to communicate with itself between nodes.  Knowing the correct token and the fact primary DA starts means that DA at least can read the shared disk acl-token. Ensure that the secondary DA (DA2) can also read the shared disk acl-token

Powered by Wolken Software