Unable to start consul after an outage of either a Fault Tolerant Data Aggregator or DA Proxy host

Products

• Network Observability
• CA Performance Management

Issue/Introduction

The Data Aggregator or Proxy host was down and offline for an extended period but when we start it back up, consul will not start

systemctl status consul shows entries like:

agent: startup error: error="refusing to rejoin cluster because server has been offline for more than the configured server_rejoin_age_max (168h0m0s) - consider wiping your data dir

Cause

By default, consul will not connect to other cluster members if it has been over 7 days.  This is both the default and maximum setting of server_rejoin_age_max and it cannot be disabled

Resolution

To get this consul service to start, you will need to remove the server_metadata.json file from within the consul data directory.  The default paths for this are as follows"

Data Aggregator: /opt/IMDataAggregator/consul/data/server_metadata.json

DA Proxy: /opt/CA/daproxy/data/server_metadata.json

After this file is removed, you will need to restart the consul service.  This will need to be done for all consul hosts that have been offline for more than 7 days:

systemctl restart consul