NSX-T cloud on Avi LB displays an error "Valid Vcenter config doesn't exists in cloud" with down/red status

Products

• VMware Avi Load Balancer

Issue/Introduction

• This error is displayed on NSX-T cloud if there's any issue with configuration related issue.
• Similar issue can also occur with vCenter cloud 
• This issue can occur if the Content library configured on NSX-T cloud was deleted and no longer available on vCenter.

Environment

• Avi version: 22.1.x/30.1.x/30.2.x with NSX-T cloud

Cause

• Content library configured on vCenter config for NSX-T cloud on Avi load balancer does not exist, due to which the cloud status displays an error "Valid Vcenter config doesn't exists in cloud"
• Below error is noticed in cc_agent_go_<Cloud Name>.log 

  2024-10-16T17:49:06.986Z        INFO    nsxtlib/nsxt_lib.go:308 Total TZs retrieved 12
    2024-10-16T17:49:06.986Z        INFO    nsxtlib/nsxt_lib.go:324 &{ID:39663e34-b02e-4fd8-a922-0ee6a7a33184 Name:GM:wedvcs12-tz-edge-public Type:VLAN_BACKED Path:/global-infra/sites/GQ-wedvcs12/enforcement-points/default/transport-zones/39663e34-b02e-4fd8-a922-0ee6a7a33184}
    2024-10-16T17:49:10.474Z        INFO    nsxt/nsxt_agent.go:1206 [We-ALB-Mgmt] Updating segmentruntime object for network network-6c23e25d-33a2-4113-901a-36609151872c
    2024-10-16T17:49:14.258Z        INFO    nsxt/nsxt_agent.go:1206 [We-ALB-Data-VIP] Updating segmentruntime object for network network-6f12de00-c6cb-42e6-829e-8668aca57f0b
    2024-10-16T17:49:14.801Z        ERROR   vcenterlib/vcenter_image.go:117 [10.x.x.x] 'GET' request to API 'content/library/id:4b59b984-5345-4e98-aba3-2e158ddb67gg'; Status: ''; Code: '404'
    2024-10-16T17:49:14.801Z        ERROR   vcenterlib/vcenter_handler.go:557       [10.x.x.x] 'GET' request to API 'content/library/id:4b59b984-5345-4e98-aba3-2e158ddb67gg'; Status: ''; Code: '404'
    2024-10-16T17:49:14.801Z        ERROR   nsxt/nsxt_agent.go:566  [10.x.x.x] 'GET' request to API 'content/library/id:4b59b984-5345-4e98-aba3-2e158ddb67gg'; Status: ''; Code: '404'
    2024-10-16T17:49:14.801Z        ERROR   nsxt/nsxt_agent.go:678  Valid Vcenter config doesn't exists in cloud
    2024-10-16T17:49:14.803Z        ERROR   cloudagent/discovery.go:116     Skipping CloudRuntime refresh and SE reconcile as the agent is not ready, error: Valid Vcenter config doesn't exists in cloud

• From the above error, 'GET' request to API 'content/library/* fails with 404 response code. A 404 HTTP response code indicates that a requested page or resource is not available.
• An Event such as below will be triggered 

   "events": [
                    {
                        "report_timestamp": 1729094954,
                        "obj_type": "CLOUDCONNECTOR",
                        "event_id": "VCENTER_ACCESS_FAIL",
                        "module": "CLOUD_CONNECTOR",
                        "internal": "EVENT_EXTERNAL",
                        "context": "EVENT_CONTEXT_SYSTEM",
                        "obj_uuid": "cluster-1b4de2e4-78a1-47ba-bda0-81501b21569f",
                        "obj_name": "We-ALB",
                        "event_details": {
                            "nsxt_info": {
                                "cc_id": "cloud-a9bbc65c-0a04-4d86-8a64-4cba383ffd94",
                                "reason": "'GET' request to API 'content/library/id:4b59b984-5345-4e98-aba3-2e158ddb67gg'; Status: ''; Code: '404'"
                            }
                        },
                        "event_description": "VCenter connection setup failed - reason 'GET' request to API 'content/library/id:4b59b984-5345-4e98-aba3-2e158ddb67gg'; Status: ''; Code: '404'",
                        "event_pages": [
                            "EVENT_PAGE_CNTLR",
                            "EVENT_PAGE_ALL"
                        ],
                        "ignore_event_details_display": false,
                        "is_security_event": false,
                        "tenant_name": "",
                        "tenant": "admin",
                        "related_refs": [
                            "/api/cluster/cluster-1b4de2e4-78a1-47ba-bda0-81501b21569f",
                            "/api/cloud/cloud-a9bbc65c-0a04-4d86-8a64-4cba383ffd94"
                        ]
                    }
                ]

Resolution

• On vCenter, navigate to the "Content Libraries" option within the vSphere Client menu, which can be accessed by clicking the hamburger menu in the top right corner and selecting "Content Libraries" from the list; this will display all available content libraries within your vCenter environment. Validate whether any of the listed Content Library is configured on Avi NSX-T cloud.
• From Avi UI > Navigate to vCenter Server config on NSX-T cloud and update the Content Library as required, the existing Content Library may no longer be available on vCenter. 
• There should be no impact to production environment when the content library is updated on vCenter/NSX-T cloud, but preferable perform this update during off business hours.