search cancel

Fault Tolerant Data Aggregator Maintenance Request is ignored

book

Article ID: 227212

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

When setting a Fault Tolerant Data Aggregator to Maintenance nothing changes.

Fault Tolerant Data Aggregator fails to change to Maintenance.

When setting a Fault Tolerant Data Aggregator to Activate nothing changes.

Fault Tolerant Data Aggregator fails to change to Activate.

We run the "./dadaemon maintenance" or "./dadaemon activate" command but the state never changes in the PC Portal web UI. We also see no messages in the consul-ext.log file on the Data Aggregator the state change is made on. This indicates the request isn't making to the DA Proxy host thus it doesn't return the request for the DA to make the state change.

A proxy was recently set up in the environment between the Fault Tolerant Data Aggregators and the Fault Tolerant Proxy server hosts.

Proxy configurations can normally be identified with the command "echo $http_proxy". If this returns a host, there is a proxy configured. If it returns nothing it's unlikely a proxy is involved in the problem.

An alternative that might also show whether or not proxies are configured is "env | grep -i proxy".

Environment

All supported DX NetOps Performance Management releases r21.2.5 and older

Cause

The dadaemon script is not set up to work properly in the presence of a proxy between the Fault Tolerant Data Aggregators and the Fault Tolerant Proxy server hosts.

Resolution

This was resolved by engineering through defect DE519171. It is resolved in the r21.2.6 release of DX NetOps Performance Management. Upgrade to r21.2.6 or newer releases for a permanent resolution.

Until this is fixed in a new release there are two options to resolve the issue in releases r21.2.5 and older.

  1. Set the following on the Fault Tolerant Data Aggregators and DA Proxy hosts to disable proxy host usage for localhost requests.
    1. export no_proxy=localhost
  2. Modify the dadaemon script to work around the proxy host presence. See below for the changes necessary.

Notes and Tips:

  • The default location is /opt/IMDataAggregator/scripts
  • Before making the change make a backup copy of the dadaemon script. Can use the command:
    • cp dadaemon dadaemon.orig<date>
  • Make the changes to the dadaemon script on both Fault Tolerant Data Aggregators.
  • Ensure the changes remain post upgrade when the script is updated/replaced.

Make the following changes to the /opt/IMDataAggregator/scripts/dadaemon script. Note the addition of the 

  • Enabled State Change CURL:
    • Change: eval "curl $CS_HEADER --silent $CS_CURL_OPT --request PUT --data 'ENABLED' $CS_PROTOCOL://localhost:$da_consul_port/v1/kv/control/$HOST_NAME" > /dev/null
    • To: eval "curl $CS_HEADER --silent $CS_CURL_OPT --noproxy localhost --request PUT --data 'ENABLED' $CS_PROTOCOL://localhost:$da_consul_port/v1/kv/control/$HOST_NAME" > /dev/null
  • Maintenance State Change CURL:
    • Change: eval "curl $CS_HEADER --silent $CS_CURL_OPT --request PUT --data 'MAINTENANCE' $CS_PROTOCOL://localhost:$da_consul_port/v1/kv/control/$HOST_NAME" > /dev/null
    • To: eval "curl $CS_HEADER --silent $CS_CURL_OPT --noproxy localhost --request PUT --data 'MAINTENANCE' $CS_PROTOCOL://localhost:$da_consul_port/v1/kv/control/$HOST_NAME" > /dev/null:
  • Remote UUID curl:
    • Change: REMOTE_UUID=$(eval "curl $CS_HEADER $CS_CURL_OPT $CS_PROTOCOL://localhost:$CS_PORT/v1/kv/failover/daservice/status?raw | cut -d':' -f1")
    • To: REMOTE_UUID=$(eval "curl $CS_HEADER $CS_CURL_OPT --noproxy localhost $CS_PROTOCOL://localhost:$CS_PORT/v1/kv/failover/daservice/status?raw | cut -d':' -f1")
  • Status check curl:
    • Change: STATUS=$(eval "curl $CS_HEADER $CS_CURL_OPT $CS_PROTOCOL://localhost:$CS_PORT/v1/kv/control/$HOST_NAME?raw")
    • To: STATUS=$(eval "curl $CS_HEADER $CS_CURL_OPT --noproxy localhost $CS_PROTOCOL://localhost:$CS_PORT/v1/kv/control/$HOST_NAME?raw")
Powered by Wolken Software