"Unable to upload the certificate. The cluster is in inconsistent state." in Aria Operations
search cancel

"Unable to upload the certificate. The cluster is in inconsistent state." in Aria Operations

book

Article ID: 414595

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • In the Admin UI (https://AriaOpsFQDNorIP/admin), in System Status > SSL Certificate (Certificate icon in the Top Right) > Install New Certificate > Browse... and when selecting a .pem file, the form displays a red underline under the file name without any error message.
    Hovering the mouse cursor over the file name field, a tooltip displays with a message about the reason why the file can not be applied:
    "Unable to upload the certificate. The cluster is in inconsistent state."



  • The casa.db.script file shows that the "remove_node_state" is in a "FAILED" state.

    SSH to the primary node and run the following command to verify:
    sed -nre "/clusterMembership/ s/^[^']+'([^']+)','([^']+)'.*/\2/p" /storage/db/casa/webapp/hsqldb/casa.db.script | python -m json.tool | grep remove_node_state
    This will be the output:
        "remove_node_state": "FAILED",

Environment

Aria Operations 8.x

Cause

The casa.db.script file didn't update properly after a failed attempt of removing a node in the cluster.

Resolution

Follow this procedure to amend the casa.db.script file and set the "remove_node_state" to a consistent state:

  1. SSH as root to the primary node and bring the cluster Offline with this command:
    $VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py offline-cluster "Amending casa.db.script"
    Wait until it completes in the admin UI.
  2. Take a VM snapshot on every node of the cluster to be able to revert if any issue.
  3. SSH as root on all the nodes of the cluster and stop the Casa service on every node:
    service vmware-casa stop
  4. Make a copy of the casa.db.script file on every node of the cluster to be able to revert if needed:
    cp /storage/db/casa/webapp/hsqldb/casa.db.script /storage/db/casa/webapp/hsqldb/casa.db.script.bak
  5. Edit the casa.db.script on every node:
    vi /storage/db/casa/webapp/hsqldb/casa.db.script
    Locate the "remove_node_state": "FAILED" and change it to "remove_node_state": "NONE"
    /wq! to save and exit.
  6. Verify the change:
    sed -nre "/clusterMembership/ s/^[^']+'([^']+)','([^']+)'.*/\2/p" /storage/db/casa/webapp/hsqldb/casa.db.script | python -m json.tool | grep remove_node_state
    This will be the output:
        "remove_node_state": "NONE",
  7. Start the Casa service on every node:
    service vmware-casa start
  8. Bring the cluster Online by running this command on the primary node:
    $VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py init-cluster
    Wait until the cluster is Online in the admin UI.
  9. Proceed Installing the new certificate.

Additional Information

Powered by Wolken Software