Identifying the root cause of vRealize Automation 7.x provisioning failures

Products

• VMware Aria Suite

Issue/Introduction

This article focuses on helping you identify the root cause or point of failure for provisioning tasks in vRealize Automation 7.x.  The focus is on later releases of 7.x but should be valid for most 7.x versions.

Environment

VMware vRealize Automation 7.x

Resolution

Troubleshooting Methodology and Best Practices

• Review the error messages starting with the error listed in the deployment failure itself.  
• Next you can review additional logs under Administration, Monitoring, Log for more information.
• If root cause is still elusive, try to reduce complexity of your blueprint to a simple provision of an object to an endpoint.  This will help you identify if the problem is within the blueprint or if it's at the endpoint or endpoint configuration.
• If the issue appears to be occurring within the Event Broker component, review the logs under Administration, Events, Event Logs.
• When possible, try the same provisioning outside of vRA (directly on the endpoint) to narrow down the source layer of the failure.
• Review endpoint configuration in Infrastructure and connectivity using the 'Test Connection' option.  Check that data collection is completing properly.

Log Analysis

vRA logs span multiple servers and services. As such, it is easier to perform deeper dive log reviews if you are using a log aggregator like vRealize Log Insight or exporting a log bundle and performing a search throughout the captured files for the errors. 

Full logs for vRealize Automation 7

https://kb.vmware.com/s/article/2141175

Logs most important for provisioning analysis

• vRA Server Logs (appliances):  /storage/log/vmware/vcac/catalina.out
• IaaS Manager Server Logs (Windows):  C:\Program Files (x86)\VMware\vCAC\Server\Logs
• IaaS Repository Logs (Windows):  C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\Repository

Finding all events related to a provisioning request

1. First you will want to collect the context ID for the request by searching for the following text strings in the IaaS Repository logs for the time the machine was first requested: MachineRequest.  In this log event you'll find a 'context=' string.
2. Perform a new search across all vRA logs for the context id from above.  This will give you a complete list of all events related to the machine request.  You should be able to review these for the exception or failure causing the provision to fail.

Log Analysis Tips

For reviewing logs using the command line the following two methods can help:

• Linux: grep –color=always -RHni “contextID” ./* | less -R
• PowerShell / Windows: $ContextID = gc <filename.log> | select-string “contextID”

Interop Log Analysis

Event Broker/ vRO failures

• If your provisioning is failing due to extensibility/ vRO workflow failures, you can log into vRO (using the account configured in vRA) and review the failed workflows for logic or other possible causes.  This is the easiest method for identifying the failures.
• The embedded vRO instances retain the context ID from vRA so when reviewing the logs you can use the context ID to then collect the related workflow ID's.  These workflow ID's can be searched to show all events related to the failing workflow.

NSX Provisioning Failures

• If your failure is related to provisioning of NSX components, you can find the full name of the NSX resource that is being attempted from the vRA logs using the string: 'CompID: [NSX'
• Using the full name of the NSX resource from above, search the NSX Manager logs for more details related to the objects being created.

AWS Provisioning Failures

• Provisioning of AWS operates the same as standard vSphere Provisioning so the above methods and best practices still apply.  
• When reviewing logs, be aware that the IaaS DEM services are brokering communication to your AWS resources so you will likely find the root cause there.

Azure Provisioning Failures

Provisioning of Azure resources function from vRO and not from vRA Infrastructure services.  For identifying the root cause, it helps to log into the vRO instance (using the account configured in vRA) and review the worklfows runs.  Note there are two places where workflows are stored:

• Library\vRealize Automation\Endpoint\Azure - This is where all of the workflows are located for vRealize Automation's native Azure functionality. Used for vRA Endpoints.
• Library\Azure\ - This is where all XaaS workflows are located. These are used when you add a vRO Endpoint for Azure and use custom XaaS blueprints and resources.

Other Resources