vRealize Automation provisioning requests fail immediately with 503 error

Products

• VMware Aria Suite

Issue/Introduction

• vRealize Automation provisioning request fail immediately.
• Request tab shows a generic 503 error.
• Catalina.out shows messages similar to the below:

2017-09-14 10:31:04,713 vcac: [component="container-service" priority="WARN" thread="tomcat-http--13" tenant="msci"] com.vmware.vcac.platform.service.rest.config.RestServiceReadinessFilter.preHandle:55 - HTTP request received before service was ready, 503 returned. URL: /container-service/api/callbacks/composition/blueprintenhancer, Method: POST, Request: SecurityContextHolderAwareRequestWrapper[ FirewalledRequest[ org.apache.catalina.connector.RequestFacade@467a31fd]]
2017-09-14 10:31:04,714 vcac: [component="cafe:composition-service" priority="ERROR" thread="queue-pool-executer-2" tenant="msci" context="jal4o1bC" parent="je9MnMP9" token="nmtgp8yY"] com.vmware.vcac.composition.service.impl.RequestServiceImpl.enhanceBlueprint:290 - Error during dynamic enhancement of blueprint request [f94590be-0cce-44a9-b34e-6b52fffd1578] org.springframework.web.client.HttpServerErrorException: 503

• One or more Xenon container service(s) may not be running within the vRealize Automation cluster

Note:​ Simple deployments with a single appliance may also be impacted.

Environment

VMware vRealize Automation 7.x

Resolution

A reboot can typically resolve the xenon service registration / startup issue.

If a reboot cannot be performed:

1. Manually start the xenon service from the vRealize Automation virtual appliance management interface (VAMI) over port 5480
2. Test provisioning of blueprints