Platform CA Refresh Status Displays "Completed With Errors" for AVI Endpoints

search cancel

Platform CA Refresh Status Displays "Completed With Errors" for AVI Endpoints

book

Article ID: 450606

calendar_today

Updated On:

Products

VMware vDefend Firewall with Advanced Threat Prevention VMware vDefend Firewall VMware Avi Load Balancer

Issue/Introduction

Occasionally, the Platform CA Refresh Status can report that it has "completed with errors" for AVI Controllers, even though the refresh workflow completed successfully on the AVI endpoints. Because we cannot completely rule out a real issue occurring during the CA refresh workflow, you must verify that these AVI endpoints have had their certificates refreshed successfully to avoid service interruption. Note that if no errors occurred on any endpoints and this is simply a case of misreporting, all endpoints will remain connected. However, we cannot rule out other underlying issues based solely on this error message.

Environment

Licensing Hub 2.0

Cause

Occasionally, site-service restarts induced by the platform CA refresh interrupt the internal AVI certificate rotation workflow progress. Because the new pod is unable to restart the workflow, trust-manager cannot receive the actual workflow status and eventually times out.

These error messages can raise concern, as we cannot rule out a genuine problem with the certificate refresh workflow. If you see this error message in the Certificate UI, verify whether it is a false negative reporting issue or an actual failure.

Detailed logs are available in the trust-manager logs, where you can see trust-manager's RolloutStatus getting stuck in the POST_SWAP_CALLBACK status for up to 3 hours.

To view these logs:

SSH to the Licensing Hub Appliance using sysadmin credentials.

Get the trust-manager pod name by running:

kubectl get pods -n nsxi-platform | grep trust-manager

View the pod logs by copying the pod name from the output and running:

kubectl -n nsxi-platform logs <pod-name-copied>

(Optional) You can append | grep <search-term> to filter for specific log entries.

The following log lines may repeatedly occur for up to 3 hours:

<182>1 2026-07-29T19:49:12.407Z trust-manager-75968b4548-424q6 SSP pool-15-thread-1 SSP [ssp@4413 comp="SSP" level="INFO" logger="TrustRolloutStatusServiceImpl" msgID="SSP" s2comp="service" subcomp="trust-manager-core" threadName="pool-15-thread-1"] Saved Status: TrustRolloutStatusDao(super=Resource(id=844a85dc-424b-48f2-b3f8-1d0297bf344f, displayName=TRUST_ROLLOUT_STATUS, description=null, createUser=null, lastModifiedUser=null, createTime=0, lastModifiedTime=1785354294286, resourceType=null, systemOwned=false, protection=null, revision=11), rolloutType=PLATFORM_CA, updatedCertificateId=null, overallStatus=IN_ROTATION, currentState=POST_SWAP_CALLBACK, progressPercent=72, rolloutTimeout=1785959094286, originalPlatformCAId=11995a1a-8f74-477b-b74c-8865f55a1707, startTime=1785354294286, taskId=94345b2f-42c3-4cd3-9a77-e7246d2a1386, originalIntermediateIds=[], updatedIntermediateIds=[], informationalMessages=[], verticalStatus=[VerticalTrustRolloutStatusDao(verticalId=22c794a7-4193-4fa2-ab59-031965d2e559, featureName=AVI, nodes=[TrustRolloutNodeStatusDao(nodeStatusId=76b41bff-9506-4a15-9c87-17f1fc148777, nodeDisplayName=0983adc1-9056-4a15-8218-9a716415f71c, nodeId=0983adc1-9056-4a15-8218-9a716415f71c, updateStatus=PENDING, errorMessage=null)], status=PENDING), VerticalTrustRolloutStatusDao(verticalId=90431cb2-49ef-48a8-a3c0-3bfae7b0fee2, featureName=SSP, nodes=[], status=COMPLETED_SUCCESSFULLY), VerticalTrustRolloutStatusDao(verticalId=09a09c00-0361-4fc1-8555-3b3821816208, featureName=NSX, nodes=[], status=COMPLETED_SUCCESSFULLY)], verticalsInvolved=[SSP, AVI, NSX])

When the platform CA refresh eventually completes, navigate to https://<licensing-hub-fqdn>/ssp/trust/trust-rollout-status in your browser and examine the result. If the AVI vertical shows a status of COMPLETED_WITH_FAILURES, it is likely a case of erroneous reporting for the AVI Controller.

Note: Other verticals such as NSX and SSP may show actual errors that require investigation. The AVI report, however, may simply be a case of false-negative reporting.

Resolution

If you verify that the AVI vertical is experiencing issues with trust rollout status, you can retrieve more details by making a specific query via the Licensing Hub API:

curl -u 'admin:<password>' -H "Content-Type: application/json" -H "Accept: application/json" -v -k -X POST "https://<lh fqdn>/ssp/trust/trust-rollout-status" -d '{"vertical_names": ["AVI"], "include_status": ["FAILED"]}'

This API returns all endpoints that were reported as a failure to complete trust rollout. For example:

{
  "rollout_type": "PLATFORM_CA",
  "informational_messages": [],
  "progress_percent": 100,
  "vertical_status": [
    {
      "vertical_key": "AVI",
      "status_summary": {
        "success_node_count": 0,
        "failed_node_count": 1,
        "pending_node_count": 0
      },
      "success_nodes": [],
      "failed_nodes": [
        {
          "node_display_name": "130ce81a-7f11-4649-8f5d-f46360931398",
          "node_id": "130ce81a-7f11-4649-8f5d-f46360931398",
          "update_status": "FAILED",
          "error_message": "Cert rotation could not be completed during the allotted time. Nodes that did not receive the updated certificates will have to be reconnected manually."
        }
      ],
      "rollout_result": "COMPLETED_WITH_FAILURES"
    }
  ],
  "overall_status": "COMPLETED_WITH_FAILURES",
  "start_time": 1785495225556
}

Verify that the trust rollout completed successfully by checking whether the individual AVI Controllers listed above have up-to-date certificates. First, collect the 3 certificate chains currently used by Licensing Hub SSP in PEM encoding using the following commands:

curl -k -u 'admin:<licensing hub password>' https://<licensing hub fqdn>/ssp/trust-management/certificates?used_by=NAPP_PLATFORM_CA
curl -k -u 'admin:<licensing hub password>' https://<licensing hub fqdn>/ssp/trust-management/certificates?used_by=NAPP_PLATFORM_INGRESS
curl -k -u 'admin:<licensing hub password>' https://<licensing hub fqdn>/ssp/trust-management/certificates?used_by=SSP_PLATFORM_EGRESS

Example Licensing Hub platform certificates of interest:

$ curl -k -u 'admin:<licensing hub password>' https://<licensing hub fqdn>/ssp/trust-management/certificates?used_by=NAPP_PLATFORM_CA
    {
      "used_by": "NAPP_PLATFORM_CA",
      "uuid": "a7fcb764-f670-4d31-91aa-89819507ffa5",
      "display_name": "SSP Platform CA-2",
      "alias": "napp_platform_ca",
      "pem_encoded": "-----BEGIN CERTIFICATE-----\CERTIFICATE-CONTENTS\-----END CERTIFICATE-----",
      "has_private_key": true,
      "type": "CERTIFICATE_CA",
      "category": "PLATFORM_CERTIFICATE",
      "ready": true
    }

$ curl -k -u 'admin:<licensing hub password>' https://<licensing hub fqdn>/ssp/trust-management/certificates?used_by=NAPP_PLATFORM_INGRESS
    {
      "used_by": "NAPP_PLATFORM_INGRESS",
      "uuid": "9d9ae49b-d39d-48a0-a2f1-9c3042857d8f",
      "display_name": "SSP Platform Ingress",
      "alias": "napp_platform_ingress",
      "pem_encoded": "-----BEGIN CERTIFICATE-----\CERTIFICATE-CONTENTS\-----END CERTIFICATE-----",
      "has_private_key": true,
      "type": "CERTIFICATE_SIGNED",
      "category": "PLATFORM_CERTIFICATE",
      "ready": true
    }

$ curl -k -u 'admin:<licensing hub password>' https://<licensing hub fqdn>/ssp/trust-management/certificates?used_by=SSP_PLATFORM_EGRESS
    {
      "used_by": "SSP_PLATFORM_EGRESS",
      "uuid": "746d04a0-f313-4c8f-acec-06fe05c7dbe7",
      "display_name": "SSP Platform Egress",
      "alias": "ssp_platform_egress",
      "pem_encoded": "-----BEGIN CERTIFICATE-----\CERTIFICATE-CONTENTS\-----END CERTIFICATE-----",
      "has_private_key": true,
      "type": "CERTIFICATE_SIGNED",
      "category": "PLATFORM_CERTIFICATE",
      "ready": true
    }

Next, retrieve the platform certificates known to the AVI Controller using the ssp-info APIs (this includes Licensing Hub's ingress, egress, and CSR certificates signed by Licensing Hub's CA).

Check the AVI Controller Version:

curl -k -u 'admin:<password>' -X GET https://<avi-controller>/api/cluster/version

Get the version from the output:

{
  "Product": "controller",
  "Version": "32.1.4",
  "ProductName": "Avi Cloud Controller"
}

Retrieve the SSP Trust Information from the AVI Controller: Using the version acquired above, run:

curl -k -u 'admin:<password>' -H 'X-Avi-Version: 32.1.4' https://<avi-controller-ip>/api/ssp

From the output, locate the section associated with your Licensing Hub (if multiple SSPs are associated with the controller). In the results, verify the following:

client_cert should exactly match Licensing Hub's SSP_PLATFORM_EGRESS certificate PEM.
ingress_cert should exactly match Licensing Hub's NAPP_PLATFORM_INGRESS certificate PEM.
avi_client_cert (the last certificate PEM in the chain) should match Licensing Hub's NAPP_PLATFORM_CA certificate PEM.

Example output demonstrating matching certificates:

{
  "count": 1,
  "results": [
    {
      "_last_modified": "1785495502681243",
      "avi_client_cert": "-----BEGIN CERTIFICATE-----\CERTIFICATE-CONTENTS\-----END CERTIFICATE-----",
      "client_cert": "-----BEGIN CERTIFICATE-----\CERTIFICATE-CONTENTS\-----END CERTIFICATE-----\n",
      "feature": "SSP_CENTRAL_LICENSING_SERVICE",
      "hostname": "hostname.example.net",
      "ingress_cert": "-----BEGIN CERTIFICATE-----\CERTIFICATE-CONTENTS\-----END CERTIFICATE-----",
      "name": "df44fea7-a9be-4767-a7b1-ccaeba24380b",
      "resources": {
        "avi_client_cert_ref": "https://192.168.1.10/api/sslkeyandcertificate/sslkeyandcertificate-0e990295-c45c-46a2-b675-d785ee94ff70",
        "client_cert_auth_mapping_profile_ref": "https://192.168.1.10/api/authmappingprofile/authmappingprofile-b45b7e03-fffe-468a-8bd5-98a386661da2", "client_cert_auth_profile_ref": "https://192.168.1.10/api/authprofile/authprofile-ab4dc659-48e0-427b-9e5e-ef72be908ae3", "client_cert_ref": "https://192.168.1.10/api/sslkeyandcertificate/sslkeyandcertificate-2a3b152a-1019-4e8f-a02b-c2ad6f8ed558" }, "status": "SSP_STATUS_ACTIVE", "tenant_ref": "https://192.168.1.10/api/tenant/admin", "url": "https://192.168.1.10/api/sspinstance/sspinstance-932b6f09-2fae-44fe-b8db-344105efc5d3", "uuid": "sspinstance-932b6f09-2fae-44fe-b8db-344105efc5d3" } ] } 

Conclusion:

If all certificates match those in Licensing Hub, the trust rollout for that AVI Controller endpoint was successful (confirming a false-negative reporting error).

If they do not match, wait 10–20 minutes and verify again using the same steps. If the mismatch persists for an extended period, off-board and re-onboard the AVI endpoint to restore trust.

Feedback

thumb_up Yes

thumb_down No