Error "minorErrorCode\":\"invalid_grant\",\"message\":\"Invalid refresh token\",\"stackTrace\":null}" when managing Encryption Management Service VMs

Products

• VMware Cloud Director

Issue/Introduction

• Encryption Management Failure on vCloud
• Blocking Access to Encrypted VMs
• The following error is seen in byok-error.log:
  • time="####-##-###########" level=error msg="cloud director endpoint https://<VCD>:443/oauth/provider/token failed: {\"error\":\"invalid_grant\",\"error_description\":\"Invalid refresh token\",\"error_uri\":null,\"minorErrorCode\":\"invalid_grant\",\"message\":\"Invalid refresh token\",\"stackTrace\":null}"
    at gitlab.eng.vmware.com/core-build/ucd-addon-byok

Environment

• VMware Cloud Director 10.5.1.1
• VMware Cloud Director Encryption Management 1.2.0

Cause

The primary cause of the authentication failures is a VCD configuration parameter, vcloud.tokens.serviceAccountTokenRotation.enabled, which is set to true by default. This setting enforces a security policy where every request for a new access token also generates a new request token

Resolution

To resolve this issue, we first to regenerate the refreshToken manually and then we can set the vcloud.tokens.serviceAccountTokenRotation.enabled flag to false, so that the request token will no longer rotate. To set this:

• From the BYOK machine, mount the VMware Cloud Director Encryption Management ISO file using command-line interface and navigate to the cli>your_operating_system folder:
  • ./vcdemctl configure refresh-token --config /etc/config.yaml --host <VCD Address> --username <username> --password <password> --insecure

 

• SSH to a VCD Database cell (Primary or Standby) and run the following command:
  • /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n "vcloud.tokens.serviceAccountTokenRotation.enabled" -v false

 

• Run the following command to verify this change:
  • /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n "vcloud.tokens.serviceAccountTokenRotation.enabled" -l