DA Proxy after upgrade - Unexpected response code: 403 (ACL not found) and 500 (No cluster leader)

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

After only upgrading the DA Proxy host from 21.2.1 to 21.2.5, the following error is displayed in the following command line outputs:

$ cat /DASharedRepo/data/acl-token.properties

Take note of the token and use it on the next syntaxes.

$ cd /opt/CA/daproxy/bin

$ ./consul members -token=<token>

Error retrieving members: Unexpected response code: 403 (ACL not found)

$ /opt/CA/daproxy/bin/consul operator raft list-peers --token=<token>

Error getting peers: Failed to retrieve raft configuration: Unexpected response code: 500 (No cluster leader)

$ ./consul kv get -recurse -token=<token>

Error querying Consul agent: Unexpected response code: 500

 

Environment

Release : 21.2

Component : IM Data Aggregator

Cause

After upgrading the DA Proxy, have to upgrade the Fault-Tolerant Data Aggregators.

1. Upgrade DA Proxy

2. Upgrade DA1

3. Upgrade DA2

Resolution

Ensure the FT DAs are in Maintenance mode and upgrade DA1 and then DA2.

Follow the steps outlined in the following section of the PM guide:

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-netops/23-3/upgrade-dx-netops/upgrading-dx-netops/upgrade-performance-monitoring/upgrade-the-data-aggregator/upgrade-fault-tolerant-data-aggregators.html 

On a long-running initialization, you can run this to watch Attribute loading which is generally the longest process.

$ tail -f karaf.log | grep "Attribute loading"

This will show the entries for the same every 1 min until it shows Attribute loading complete.