EC2 Start Instances Policy Action Failed to Start instances - Server.InternalError

Products

• CloudHealth

Issue/Introduction

 

Overview:
Policy Actions can be configured to start EC2 instances using the AWS API.  After attempting to start an instance (and after receiving a good response from the AWS API call to start) a 40 minute wait timer is invoked.  This wait time is to allow for the instance state to settle and for normal state collection to complete and reflect in our platform.  After the timer runs out the instance state is checked.  If the newly started instance is not in a running state at that time then the the action is marked as "Failed" and an "Instance(s) not Started ..." message is displayed in the Action results.  This message is followed by the instance id or list of instances ids and their accounts of the failures.

 

Troubleshooting:

First check the state of the instance.  Do this in the CloudHealth application or your AWS console. 

Make sure the instance is in a running state (or was in a running state at the time of the verification check). 

Instances that are not in a Running state 40 minutes after the API start call will correctly be marked as "Failed". 

Check the change log for the asset in the platform asset report or in AWS logs.

Under certain AWS configurations the log will show that the instance has failed to start with the  message "Server.Internal.Error". 

This can occur due to the volume associated with the instance failing to attach.

Review the CloudTrail logs for this event. 

In some cases the volume may be encrypted.  If so see AWS KB articles...

https://repost.aws/knowledge-center/kms-iam-ec2-permission
https://repost.aws/knowledge-center/encrypted-volumes-stops-immediately

Apply any additional KMS permissions to the IAM role used by the account in the platform to start instances.