After we upgraded to 20.2.10, under system status in the CAPC console, we are seeing one DA only.
After upgrade to 20.2.10, under system status in CAPC console, we are not seeing either DA.
We have introduced an acl token in 20.2.10 and that token must be shared between the proxy and both DA’s for the proxy consul service to be able to communicate.
Dx NetOps Performance Management 20.2
1. Stop both DA’s and consul services
Try the maintenance command first
If this doesn’t stop the DA after 5 minutes, use systemctl
systemctl stop dadaemon;
systemctl stop activemq;
And stop the consul services
systemctl stop consul;
systemctl stop consul-ext;
2. On proxy host:
Stop the daproxy and consul services
systemctl stop consul;
systemctl stop daproxy;
Start the daproxy service;
systemctl start daproxy;
3. Rename the consul data directory on all three hosts (DA’s and proxy)
default location on DA: /opt/IMDataAggregator/consul/data
mv /opt/IMDataAggregator/consul/data /opt/IMDataAggregator/consul/data.old;
default location on Proxy: /opt/CA/daproxy/data
mv /opt/CA/daproxy/data /opt/CA/daproxy/data.old;
4. Start consul service on all three hosts
systemctl start consul
5. Confirm a leader has been selected by consul
Run on DA's and proxy:
It should return the proxy host
If it returns "", there is no leader.
So you should do "systemctl status consul" on DA proxy and DAs to see if a consul is having issues.
Resolve the issue/error, and restart consul and check leader URL.
6. If ACL was enabled and bootstrapped, the old acl master token will become invalid once all the data directories are deleted.
We will then need to bootstrap ACL again.
We will be making an HTTP PUT request to one of the DA consul servers
curl -v -X PUT -H 'Content-Type: application/xml' http://DA-HOST:8500/v1/acl/bootstrap
This request will then return a SecretId, which will be the new consul acl master token.
7. Update the acl-token.properties file in the DA shared repo with this new token.
cd to your sharedrepo, this will be named whatever you chose during the install
cp acl-token.properties acl-token.properties.original
Replace the old token with the new token from the SecretId value in the PUT command results.
8. Start the consul-ext service on both DA's
systemctl start consul-ext
9. Verify that you can now see all members on all nodes
/opt/IMDataAggregator/consul/bin/consul kv get -recurse -token=cb37db60-0088-70bd-092c-36f2e507c406 (replace with your token)