VCFA Deployment attempts with a custom workflow to add attributes to vCenter VMs is failing after upgrading to VCF 9.0.2

Products

• VCF Automation

Issue/Introduction

• Automation Deployments are failing with "Create Failed" and giving an extensibility error message similar to:
  
  Failure: Extensibility error received for topic ... failed with the following error.  Workflow run... completed with error [You can get/set custom attribute only on VC ManagedEntites (Dynamic Script Module name: ... )]
  
  
• An Orchestrator workflow which searches the inventory for a VM object by name comes back undefined.
• The next step attempts to use the undefined object and fails out with
  
  You can get/set custom attribute only on VC ManagedEntites (Dynamic Script Module name: ... )
  
  
• Further inspection shows that the previous command to obtain the VM object comes back as undefined but still passes the object to the step expecting a VM object.
• Inspecting the Orchestrator Inventory shows at least one of the vCenters in the following state:
  
  vCenter https://.../sdk (unusable: java.lang.reflect.InvocationTargetException)

Environment

VMware Cloud Foundation 9.0.2
VMware Cloud Foundation Automation 9.0.2
VMware Cloud Foundation Operations Orchestrator 9.0.2
vCenter Server 8.0.3

Cause

This happens because the custom workflow is calling to search the inventory of the entire vCenter Plugin, and at least one of the vCenters inventory is in an error state, and the expectation of the custom workflow is to pull back a VM object and pass it to be acted upon.  When the undefined object is passed, the call to act upon the object fails.

Resolution

Run the "Update a vCenter Server Instance" from the Library>Workflow menu to refresh the problematic vCenter's connection to Orchestrator.

Additional Information

If the issue reoccurs please report the problem to Broadcom support by opening a case and having a support bundle ready from VCFA, or Orchestrator if it is standalone, to review.