Intermittent Extensibility Workflow Failures (PolyglotRunnerException) during Post-Provisioning in VCF Automation 9.1.0

Products

• VCF Automation

Issue/Introduction

In VCF Automation 9.1.0, user-defined VCF Operations Orchestrator extensibility workflows executed via Event Broker Service (EBS) subscriptions experience the following issues:
This behavior is observed following an upgrade from the 9.0.2 to VCF 9.1.0.

• Workflows fail intermittently or are dropped entirely.
• Issues occur primarily when subscriptions are set to Priority 0 (Immediate Execution).
• Modification of subscriptions to lower or staggered priority levels (e.g., Priority 2 or 3) allows successful execution.
• Workflows fail with PolyglotRunnerException: Function execution returned code: 1.
• Failures are intermittent and have been observed at multiple subscription priority levels, including Priority 0 and Priority 1.
• Guest OS logs (e.g., guestcust.log or vmware-vmsvc.log) may report VIX error 3016 (VIX_E_TOOLS_NOT_RUNNING) or Win32 Error 1907 during the failure window.

 

Environment

• VCF Automation 9.1.0
• VCF Operations Orchestrator 9.1.0

Cause

The underlying cause is a timing race condition between the Guest OS initialization process (Cloud-init or Cloudbase-init) and the VCF Automation post-provisioning workflow execution.

If a workflow attempts to execute a script via Guest Ops (e.g., Invoke-VMScript) while the Guest OS agent is still performing intensive initialization tasks—such as joining a domain, applying group policies, or processing OOBE configurations—the VMware Tools RPC channel may become unresponsive or return an execution error. This race condition is more prevalent in VCF 9.1.0 due to changes in how execution threads are managed compared to previous versions.

Resolution

Workaround

1. Implement Script Retry Logic: Modify PowerShell or PowerCLI scripts to use a Try-Catch block. If the command fails, the script should sleep for 60 seconds and retry the operation (up to 3 times).
2. Add a Blocking Workflow: Insert a "Wait for Guest Initialization" workflow at the beginning of the subscription chain. This workflow should block subsequent tasks until it verifies that the initialization agent (Cloud-init/Cloudbase-init) has completed its routine.
3. Template Optimization: Temporarily disable or remove unnecessary initialization configurations from the VM template to reduce the duration of the OOBE phase and minimize the collision window.

Additional Information

Managing Event Subscriptions in VCF Automation

"Blocking subscriptions run in priority order. The highest priority value is 0 (zero). If you have more than one blocking subscription for the same event topic with the same priority level, the subscriptions run in a reverse alphabetical order based on the name of the subscription. After all blocking subscriptions are processed, the message is sent to all the non-blocking subscriptions at the same time. Because the blocking subscriptions run synchronously, the changed event payload includes the updated event when the subsequent subscriptions are notified."