vSphere functions in Aria Orchestrator stop working: ephemeral certificate lifetime is too short

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Possible symptoms:

• Scheduled workflows in Orchestrator stop working after one week with errors such as the below seen in the logs (vco-server-app.log):
  • ERROR vco [host='vco-app-########-####' thread='tokenLifetimeMonitorScheduler-1' user='' org='' trace=''] {} com.vmware.o11n.security.session.ManagedTokenRegistryImpl - Unable to convert token with id ################################
com.vmware.vcac.authentication.http.SamlAuthenticationException: Token expiration date: <DATE> is in the past.
  • ERROR vco [host='vco-app-########-####' thread='tokenLifetimeMonitorScheduler-1' user='' org='' trace=''] {} com.vmware.vim.sso.client.impl.SoapBindingImpl - SOAP fault
com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Unable to renew non-renewable token Please see the server log to find more detail regarding exact cause of the failure.
• Workflows may fail about every 90 minutes since the token is renewed without extending the expiry time:
  • INFO vco [host='vco-app-########-####' thread='http-nio-8280-exec-##' user='<username>' org='-' trace='####-##-##-##-######'] {} com.vmware.o11n.security.sso.support.SamlTokenLifetimeService - Token renewed successfully: id _####-##-##-##-######, new expiration date: <DATE1>
  • ERROR vco [host='vco-app-########-####' thread='http-nio-8280-exec-##' user='-' org='-' trace='-'] {} com.vmware.vim.sso.client.impl.SoapBindingImpl - SOAP fault
com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: EndTime: <DATE1> is not after startTime: <DATE2> Please see the server log to find more detail regarding exact cause of the failure.
• vCenters may be listed as "unusable" in Orchestrator -> Administration -> Inventory -> vCenter plugin.
• If vCenter SSO is used for authentication, logging in to Orchestrator may show the  "${message}  ${backToLoginLabel}" error screen:
  • 

 

 

Environment

Aria Automation/Orchestrator 8.18.1

Cause

Aria Orchestrator fails to renew its access token to vCenter appropriately due to ephemeral certificate expiry.

Resolution

This is a known issue which is fixed in Aria Orchestrator 8.18.1 Patch 4.

 

Workaround

Alternatively, the below steps can be followed to resolve the issue without Patch 4:

1. Install Orchestrator 8.18.1 Patch 3.
2. Add custom property "com.vmware.o11n.sso.svcaccount.ephemeral-cert-lifetime-ms" to the affected Orchestrator, as per KB 408782. 
3. Then remove the current schedule for the affected workflow and re-schedule it.

Scheduled workflows will then continue running after one week.