Error: Invalid response received while obtaining inventory/metrics for the collection status of the VCF services runtime adapter on cloud proxies in VCF Operations 9.1

Products

• VCF Operations
• VCF Automation

Issue/Introduction

• In VMware Cloud Foundation (VCF) Operations 9.1, the Cloud Proxies' collection status experiences an API failure. The Operations UI and VMSP adapter logs report the following exact error message:
  

  Invalid response received while obtaining inventory/metrics - {"messages":[{"level":"ERROR","id":"com.vmware.vcfms.api.NotFound","default":"Not Found"}]}

• As a result, the VMSP Adapter instance changes to an Error state and fails to successfully collect inventory or metrics. 
• There was a recent failed VCFA upgrade which resulted in a redeployment of VCFA 9.1
• When running the below command on the VMSP control plane, there is a VCFA 9.0 component in an unknown state.
  

  kubectl get components -A --kubeconfig=/etc/kubernetes/admin.conf

  

Environment

• VCF Operations 9.1
• VCF Automation 9.1
• VCF Automation 9.0

Cause

This issue is caused by a single verified root cause: a stale legacy VCF Automation (VCFA) 9.0 component remains on the Management VMSP cluster in an "Unknown" state following a failed upgrade and subsequent rollback attempt.

The VMSP Adapter fails when it attempts to query this non-running component during the collection cycle

Resolution

Warning: Do not run step 4 unless the 9.0.0 VCFA component is found during step 2

1. SSH into the Control Plane IP using the vmware-system-user account.

2. Identify the stale VCFA entry by executing the following command: 
   

   kubectl get components -A --kubeconfig=/etc/kubernetes/admin.conf

3. Confirm that the vcfa component is stuck in the Unknown status.

4. Safely remove the stale component from the Management VSP cluster by executing:
   

   sudo kubectl delete component vcfa --kubeconfig=/etc/kubernetes/admin.conf

5. Navigate to the VCF Operations inventory (Adapter instances).

6. Restart the VCF management services adapter instance.

7. Verify that objects and metrics resume collecting successfully without any further API errors.