"[Error code: 42100 ] - [Error Msg: Infrastructure service provider error: The action is invalid for the machine.]" in vRA 7.x

Products

• VMware Aria Suite

Issue/Introduction

This article provides resolution steps when the IaaS database and the embedded postgres database are out of sync.

Symptoms:

• Requests to expire virtual machine deployments fail, with the error similar to:
  
  [Error code: 42100 ] - [Error Msg: Infrastructure service provider error: The action is invalid for the machine.]
   
• Email notifications for the expire requests triggers approximately every two hours.
• Checking the machine details for the expiring deployment, you find that it is already expired.
• In the vRealize Automation catalina.out log file, you see entries similar to:
  
  2016-12-05 16:57:01,097 vcac: [component="cafe:catalog" priority="WARN" thread="tomcat-http--7" tenant="vsphere.local" context="5zM4rJms" token="6CerP64t" com.vmware.vcac.catalog.service.impl.RequestServiceImpl.completeRequestByProvider:594 - Request <UUID_1> with providerBindingId <UUID_2> provisioning failed with details: The following component requests failed: Machine. Exception during request callback with id <UUID_3> for item <UUID_4>. Error Message: [Error code: 42100 ] - [Error Msg: Infrastructure service provider error: The action is invalid for the machine.]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Environment

VMware vRealize Automation 7.0.x
VMware vRealize Automation 7.1.x
VMware vRealize Automation 7.2.x

Cause

This issue occurs when the IaaS database and the embedded postgres database are out of sync, putting the machine and deployment in an inconsistent state.

Resolution

This issue is resolved in vRealize Automation 7.3, available at VMware Downloads.

 

To resolve the issue if you do not want to upgrade:

 

1. Log in to the IaaS database and run the below script to get a list of all expired machines in your system:
   
   // Get expired machines from IaaS SQL server
   GO
   DECLARE @listStr NVARCHAR(max)
   
   SELECT @listStr = COALESCE(+@listStr+',', '') + ''''
                     + virtualmachineproperties.propertyvalue
                     + ''''
   FROM virtualmachine
          JOIN virtualmachineproperties
            ON virtualmachine.virtualmachineid = virtualmachineproperties.entityid
               AND virtualmachineproperties.propertyname =
                   '__iaas_request_binding_id'
   WHERE virtualmachinestate = 'Expired'
   SELECT @listStr
   GO
2. Open an SSH session to the vRealize Automation VA and log in to the embedded postgres by running the following:
    
   1. Navigate to /opt/vmware/vpostgres/current/bin
   2. Enter the command: su postgres
   3. Enter the command: ./psql vcac
       
3. Create a backup of the table getting modified by running the command:
   
   Select * into cat_resource_bkp_temp from cat_resource
   
   Note: Ensure the table cat_resource_bkp_temp is created before you proceed further.
    
4. Run the query below against vcac postgres database, by replacing <list> with the output list from step #1.
   
   // Update the deployments in Postgres that have machines that have expired as per SQL Server
   UPDATE cat_resource
   SET leasestate = 'ARCHIVED'
   WHERE id IN
          (
                 SELECT id AS parentid
                 FROM cat_resource
                 WHERE id IN
                        (
                               SELECT Cast(cafe_resource_id AS UUID)
                               FROM comp_deployment
                               WHERE prov_req_id IN
                                      (
                                             SELECT parent_id
                                             FROM comp_comprequest
                                             WHERE binding_id IN ( <list>)
                                       )
                       )
          )
   AND
   leasestate = 'ACTIVE'
5. Run the query below against vcac postgres database, by replacing <list> with the output list from step #1.
   
   UPDATE cat_resource
   SET leasestate = 'ARCHIVED'
   WHERE id IN (SELECT parent.id
   FROM cat_resource parent
   JOIN cat_resource child
   ON parent.id = child.parentresource_id
   WHERE child.NAME IN (<list>))
   
   
   For example, the query looks similar to:
   
   UPDATE cat_resource
   SET leasestate = 'ARCHIVED'
   WHERE id IN (SELECT parent.id
   FROM cat_resource parent
   JOIN cat_resource child
   ON parent.id = child.parentresource_id
   WHERE child.NAME IN ( 'VivekMn076', 'VivekMn073', 'VivekMn074' ))
   
   
   This query matches the state of expired machines in the IaaS database and machines in the Postgres database, bringing back consistency.
    
6. (Optional) After verifying that the deployments in machines_to_expire_temp have stopped sending out expire request failed notifications, drop the table machines_to_expire_temp and cat_resource_bkp_temp.

 

Note: If the queries are not run correctly, it can cause database corruption. Always take a backup to provide a stable roll-back point.