Data Aggregator nodes in FT do not start

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

Data Aggregators in Fault-Tolerant do not start.

Environment

Release : any

Component : IM Data Aggregator

Cause

The following error was found in the /opt/CA/IMDataAggregator/consul-ext/data/logs/consul-ext.log file:

2022-01-31 13:47:14 ERROR GroovyExecutor:71 - groovy execution failure: /opt/CA/IMDataAggregator/consul-ext/scripts/checkDAStatus.groovycom.ca.consul.ext.executor.GroovyExecutionException: (Error Line: 128)/opt/CA/IMDataAggregator/consul-ext/data/failover/daservice.uuid (No such file or directory)

The /opt/CA/IMDataAggregator/consul-ext/data/failover/daservice.uuid file does not exist.

Actually, the /opt/CA/IMDataAggregator/consul-ext/data/failover/ directory does not exist on both DA nodes.

The IMDataAggregator/consul-ext/data/failover/daservice.uuid is the unique ID for the DA. Each DA should have one. 

It seems somehow it got removed.

Resolution

Generated the new UUID and put it in the daservice.uuid file and restart the consul service on both DA nodes. 

1. Go to the https://www.uuidgenerator.net/  URL.

2. Use the Version 4 UUID generator. 

3. Created the /opt/CA/IMDataAggregator/consul-ext/data/failover/ directory.

4. Created the /opt/CA/IMDataAggregator/consul-ext/data/failover/daservice.uuid file and pasted the UUID from the URL above. Saved the file.

5. Changed the ownership of the /opt/CA/IMDataAggregator/consul-ext/data/failover/ directory to match the others directories owner.

6. Changed the /opt/CA/IMDataAggregator/consul-ext/data/failover/daservice.uuid file permission to 777.

7. Generated a new UUID for the other DA node and performed the same steps (1-6) as each DA node must have a unique UUID.

Also, recreated the acl-token file 

8. Stopped the consul and consul-ext services on each DA node.

$ service consul stop

$ service consul-ext stop

8. Stopped the consul service on the DA proxy.

$ service consul stop

9. On the DA proxy, reset the consul data.

$ mv /opt/CA/daproxy/data

10. On the DA nodes, reset consul data.

$ mv /opt/IMDataAggregator/consul/data

11. On the DA proxy, run:

$ service consul start

12. On the DA nodes, run:

$ service consul start

13. On one of the DAs, run:  curl http://127.0.0.1:8500/v1/status/leader   and confirm a leader is returned (aka IP:8300).

14. Once a leader is available, run on one of the DAs: curl -X PUT  http://127.0.0.1:8500/v1/acl/bootstrap  and it should return a body that contains a "SecretId:"

15. Take the SecretID value, create /DASharedRepo/acl-token.properties file on the DA Proxy host, and add the line.

16. Run the following syntax:

$ ./consul operator raft list-peers -token <token>

$ ./consul members -token <token>

17. Activate the DA nodes.

$ cd /opt/CA/IMDataAggregator/scripts

$ ./dadaemon activate

$ ./dadaemon status (only one DA node will be in running state, the other node will be stopped).

18. DA is now up in PC.