Proxy Agents are marked 'Down' and Provisioning, Data Collection, and Day 2 operations are failing

Products

• VMware Aria Suite

Issue/Introduction

This article describes how to resolve issues surrounding solution user expiry affecting proxy agent communication.

Symptoms:

• IaaS vSphere agents are marked as Down in Infrastructure tab.
• Provisioning, Data collections, and Day2 operations are failing.
• On the respective IaaS node(s) C:\Program Files (x86)\VMware\vCAC\Agents\agent_name\logs\vSphereAgent.log contains errors similar to:

  Exception occurred when retrieving work item from Manager Service

• On the Primary IaaS web server within Internet Information Services > Server > Certificates,  all IaaS solution users are expired.  These certificates are in the format of iaas.usr-UID
• The following error messages are seen in the vSphereAgent.log

System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. ---> System.ServiceModel.FaultException: At least one security token in the message could not be validated.

Environment

VMware vRealize Automation 7.x

Cause

The Manager Service cannot validate the Proxy Agent client certificate. The IaaS solution user has a default expiry time of 5 years. This could happen if the IaaS Solution user certificate, used by the Proxy Agent to identify against Manager service, has expired.

Resolution

Prerequisites

• Simultaneously snapshot each vRA appliance without memory. 
• Take a snapshot of each Iaas server.
• Full IaaS database backup (from Microsoft SQL Management Studio or other method).

Re-register the IaaS Solution user certificate

Run the below commands using an elevated command prompt.
Note:  Update italicized text based upon your environment.

1. Open a command prompt on the primary IaaS Web Server.
2. Change directory to the Web installation directory.  This may differ depending on your installation.

   cd C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\Cafe\

3. Backup the old vcac-config.data file:

   Rename vcac-config.data vcac-config.old

4. Run the following to collect updated certificates:

   Vcac-Config.exe GetServerCertificates -url https://vRA-Appliance-or-LoadBalancer-FQDN --FileName .\vcac-config.data

5. Run the following to register the Solution User:

   Vcac-Config.exe RegisterSolutionUser -url https://vRA-Appliance-or-LoadBalancer-FQDN --Tenant vsphere.local -cu "[email protected]" -cp Password --FileName .\Vcac-Config.data -v

Where Password is the account password

5. Run the following to move the Solution User registration into the DB: 

   Vcac-Config.exe MoveRegistrationDataToDB -d DB_Name -s Server_name -f .\Vcac-Config.data -v

   Where DB_Name is the IaaS SQL database name and Server_name is the IaaS SQL Server.
6. Login to the vRA VAMI (Accessing vRealize Automation 7.x's Virtual Appliance Management Interface (VAMI) Best Practices) and click vRA > Certificates > Actions > Reinitiate Trust.