VMware vRealize Application Services application deployment fails with the error: Unable to retrieve state due to 401 Unauthorized

Products

• VMware

Issue/Introduction

Symptoms:

• Deployment of applications through vRealize Automation (formerly known as vCloud Automation Center) 6.1.x fails in vRealize Application Services (formerly known as vCloud Application Director) 6.1.x.
• In the catalina.out file, located at /home/darwin/tcserver/darwin/logs/ in the vRealize Application Services appliance, you see entries similar to:
  
  14:40:40.592 INFO [main] [] com.vmware.darwin.agent.ssl.AgentKeystoreInitializer - Downloading certificates and amqp credentials from the server ...
  14:40:41.083 DEBUG [main] [] com.vmware.darwin.agent.ssl.BasicAuthSSLHttpClient - Attempt 1 to execute request
  14:40:41.102 DEBUG [main] [] com.vmware.darwin.agent.ssl.BasicAuthSSLHttpClient - Connection can be kept alive indefinitely
  14:40:41.102 DEBUG [main] [] com.vmware.darwin.agent.ssl.BasicAuthSSLHttpClient - Authentication required
  14:40:41.102 DEBUG [main] [] com.vmware.darwin.agent.ssl.BasicAuthSSLHttpClient - 10.56.172.203:8443 requested authentication
  14:40:41.110 DEBUG [main] [] com.vmware.darwin.agent.ssl.BasicAuthSSLHttpClient - Selected authentication options: [BASIC]
  14:40:41.113 DEBUG [main] [] com.vmware.darwin.agent.ssl.BasicAuthSSLHttpClient - Attempt 2 to execute request
  14:40:41.730 DEBUG [main] [] com.vmware.darwin.agent.ssl.BasicAuthSSLHttpClient - Connection can be kept alive indefinitely
  14:40:41.731 DEBUG [main] [] com.vmware.darwin.agent.ssl.BasicAuthSSLHttpClient - Authentication succeeded
  ...
  15:02:28 CEST 2014 Agent ran before; this is a new execution of the agent, perhaps after a reboot.
  15:02:28 CEST 2014 Looking for agent properties file...
  15:02:28 CEST 2014 Found and will use /opt/vmware-appdirector/agent/appd.properties.
  15:02:28 CEST 2014 Found existing /opt/vmware-appdirector/agent/nobel-agent.jar in the VM.
  15:02:28 CEST 2014 sha256sum matched. Use existing /opt/vmware-appdirector/agent/nobel-agent.jar.
  15:02:28 CEST 2014 Using /opt/vmware-jre/bin/java
  ...
  15:23:25.311 DEBUG [flowExecutionExecutor-1] [] com.vmware.darwin.agent.ssl.X509SSLHttpClient - 10.56.172.203:8443 requested authentication
  15:23:25.332 WARN [flowExecutionExecutor-1] [] com.vmware.darwin.agent.AppAgent - Unable to retrieve state due to 401 Unauthorized; attempts left = 9.
  ...
  15:24:10.434 WARN [flowExecutionExecutor-1] [] com.vmware.darwin.agent.AppAgent - Unable to retrieve state due to 401 Unauthorized; attempts left = 0.
  ...
  15:24:10.445 DEBUG [flowExecutionExecutor-1] [] com.vmware.darwin.agent.AppAgent - Posting task state 'FAILED' for task (vCAD_RHEL6_OBIEE_VLAN170_Node1, host).

Cause

This issue occurs due to the low timeout values in VMware vRealize Application Services (formerly known as vCloud Application Director)6.0.1.

Resolution

This issue is resolved in VMware vRealize Application Services (formerly known as vCloud Application Director) 6.1, available at VMware Downloads. For more information, see VMware vCloud Automation Center 6.1 Release Notes.

If you cannot upgrade to VMware vRealize Application Services, use this workaround.

To work around the issue, increase the node-task timeout value through the use of a system property set in vRealize Application Services appliance init.d script CATALINA_OPTS.

1. SSH in to the vRealize Application Services appliance.
2. Stop tcServer by running this command:
   
   service vmware-darwin-tcserver stop
   
   
3. Run this command:
   
   sudo vi /etc/init.d/vmware-darwin-tcserver
   
   
4. Press i to edit.
5. Enter Dnodetask.timeout=75 -Dhosttask.timeout=75 at the beginning of all the other options.
6. Press the escape key, then type wq! and press enter.
7. Start tcServer by running this command:
   
   service vmware-darwin-tcserver start