Recovery of License Hub in case of crash or disaster

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

License Hub consists of a controller node and a worker node. Failure of the License Hub worker node is handled automatically by the system and no user intervention is needed.

User intervention is needed in case of License hub control node failures, which is considered License Hub failure in the context of this document. This document lists down the steps to be followed to recover from such a failure.

Environment

License Hub 5.1.2

Cause

License Hub control node VM crashes

Resolution

Proactive Recommendations

To prepare for potential service interruptions, we recommend the following at all times:

- Maintain a list of all endpoints (NSX Manager, Avi Controllers, or SSP) onboarded to each License Hub along with their IP addresses or FQDNs. This information can be obtained from the License hub UI or via API.

List Endpoints

GET https://{{license-hub-url}}/licensing/endpoints
{
  "total_result_count": 1,
  "results": [
    {
      "_create_time": 1774512965896,
      "_last_modified_time": 1774513050422,
      "_revision": 3,
      "display_name": "nsx-mgr-0",
      "id": "5b03f31f-6349-42f8-afac-178eb5cddd3d",
      "connection_info": {
        "connection_type": "DYNAMIC",
        "hostname": "ip-address"
      },
      "endpoint_type": "NSX_MANAGER",
      "status": "READY"
    }
  ]
}

- Keep records of license assignments for each of these endpoints. This information can be obtained from the License hub UI or via API.

Get License Endpoint Mappings

GET https://{{license-hub-url}}/licensing/license-endpoint-mappings
{
  "total_result_count": 2,
  "results": [
    {
      "endpoint_id": "5b03f31f-6349-42f8-afac-178eb5cddd3d",
      "id": "50c00d13-07c5-435a-a6f7-9e83a5621688",
      "license_id": "ZEUSS-00MLY-UBL6Q-NCL1Z-T4L6U"
    },
    {
      "endpoint_id": "5b03f31f-6349-42f8-afac-178eb5cddd3d",
      "id": "22eb0670-811d-48e8-99a4-71b41e943fba",
      "license_id": "ZEUSS-00MLY-UEK8X-JTAPG-51BA9"
    }
  ]
}

Recovery Steps

If the License hub crashes and is not recoverable install a new License Hub from SSPI
- Before installing the new License Hub, delete the existing License Hub instance
- Install the new License Hub
Register the new License Hub to Avi Cloud Console; please note it will appear with a different Asset ID. Register the new License Hub to the same site as the older License Hub
Make a note of all license assignments to the failed LH in the AVI Cloud console.
In Avi Cloud Console, deregister the failed License Hub to make its licenses available for reassignment.
Assign all licenses originally belonging to the failed hub to the newly registered License Hub.
Use your records (described in the 'proactive recommendations') to identify the endpoints to onboard to the new license hub.
In case steps mentioned in 'proactive recommendations' are not done, then user should manually make a note of Endpoints which were connected to License Hub instance and also note the Licenses that were assigned to each Endpoint by checking the Licenses Page inside each Endpoint.
Start onboarding endpoints one at a time. Each endpoint will first need to be force off-boarded before it can be re-onboarded to the new license hub. This can be done using scripts provided for cleaning up each type of endpoint mentioned in KB: License Hub instance cleanup from onboarded endpoints (NSX, SSP & AVI).Note that this step will remove the existing licenses from the endpoint. The customer can choose to do this step during maintenance window, to reduce the impact to the system. The licenses will be reintroduced after onboarding and reassignment of licenses from the new License. The time between force-offboarding and re-onboarding to the new license hub should be minimized to reduce the time window when the endpoint is running without licenses. When there are no license on the endpoint, it will go in enforcement-mode
Licenses can be assingned to the endpiont as a part of its onboarding, and these should match the assignment records mentioned in the 'proactive recommendations' section.
Confirm that the licenses have been assigned to the endpoints by login into the endpoint and checking the licenses.