Alert: Adapter instance object has error status on VCF object(s)

Products

• VCF Operations/Automation (formerly VMware Aria Suite)
• VCF Operations

Issue/Introduction

• Receiving an alert 'Adapter instance object has error status' triggering on VCF object(s)
• Status of the adapter instance shows as OK (8.x) or Collecting (9.x)
• Attempting validate connection for the VCF adapter instance fails with error "Request is taking longer time than expected, please try again in sometime"
• In /storage/log/vcops/logs/adapters/VcfAdapter/VcfAdapter_##.log on the managing the VCF adapter,  you see entries similar errors to:
  
  'Error occured in InventoryProcessor. Reason: Request is taking longer time than expected, please try again in sometime com.vmware.vrops.adapter.vcf.exception.VcfException: Request is taking longer time than expected, please try again in sometime'
  
  Or
  
  WARN  VcfAdapter 2866681 [ops@4413 threadId="792" threadName="Collector worker thread 22" instanceId="76"] [(76) com.vmware.vrops.adapter.vcf.auth.AuthenticationManager.warn] - Exception occurred while connecting to url <https://SDDC_URL/v1/tokens>
com.vmware.vrops.adapter.vcf.exception.VcfException: Request is taking longer time than expected, please try again in sometime
        at com.vmware.vrops.adapter.vcf.util.VcfAdapterRestClient.executeWithRetry(VcfAdapterRestClient.java:167) ~[VcfAdapter.jar:?]
        at com.vmware.vrops.adapter.vcf.auth.AuthenticationManager.getNewAccessTokenDataWithRetry(AuthenticationManager.java:61) ~[VcfAdapter.jar:?]
        at com.vmware.vrops.adapter.vcf.auth.AuthenticationManager.getNewAccessTokenDataWithoutRetry(AuthenticationManager.java:48) ~[VcfAdapter.jar:?]
at com.vmware.vrops.adapter.vcf.helper.VcfAdapterHelper.extractAdapterSettings(VcfAdapterHelper.java:142) ~[VcfAdapter.jar:?]
        at com.vmware.vrops.adapter.vcf.helper.VcfAdapterHelper.initializeAdapterConfiguration(VcfAdapterHelper.java:89) ~[VcfAdapter.jar:?]
        at com.vmware.vrops.adapter.vcf.VcfAdapter.initAdapter(VcfAdapter.java:357) ~[VcfAdapter.jar:?]
        at com.vmware.vrops.adapter.vcf.VcfAdapter.onCollect(VcfAdapter.java:202) ~[VcfAdapter.jar:?]
        at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:775) ~[vrops-adapters-sdk.jar:?]
at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:547) ~[vrops-adapters-sdk.jar:?]
        

Environment

VMware Aria Operations 8.x
VCF Operations 9.x

Cause

Slow API queries from VCF adapter in VCF/Aria Operations

Resolution

Step 1:

If there is a temporary network slowness on these nodes, reboot the VCF Operations nodes or SDDC manager nodes. If the issue persists, proceed to step.2. 

Step 2:

Important: Take snapshots of each of the VCF/Aria Operations nodes before applying any of the steps below

How to take a Snapshot of VCF/Aria Operations

1. SSH to primary node as root
2. Stop collector service: systemctl stop collector
3. Create a copy of 'config.properties' file: 
   
   cp /usr/lib/vmware-vcops/user/plugins/inbound/VcfAdapter/conf/config.properties /usr/lib/vmware-vcops/user/plugins/inbound/VcfAdapter/conf/config.properties.bkp
   
   
4. Edit the file 'config.properties' file: vi /usr/lib/vmware-vcops/user/plugins/inbound/VcfAdapter/conf/config.properties
5. Modify the API connection timeout from current value to 20. For ex:
   
   Before: "api.connection.timeout=30"
   After: "api.connection.timeout=120"
   
   
6. Save the file and start the collector service: systemctl start collector
7. Repeat these steps on all collectors that manage the VCF adapters.