VIC containers fail to create or run with "Permission to perform this operation was denied"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
The docker-client fails to run the container with a volume reporting the following error:

Error response from daemon: Server error from portlayer: ServerFaultCode: Permission to perform this operation was denied..

The docker-personality log reports messages similar to the following:

Jul 30 2018 22:05:25.956Z ERROR Handler for POST /v1.24/containers/create returned error: Server error from portlayer: ServerFaultCode: Permission to perform this operation was denied.
 

Note: The docker-personality.log is located in /var/log/vic/docker-personality.log on the VCH endpoint VM. The log can be accessed in the following ways:
Browser: https://VCH-IP-FQDN:2378/logs/docker-personality.log
Shell: Enable the shell using "vic-machine debug", see link below. Once enabled connect a SSH session and run 'vi /var/log/vic/docker-personality.log'. 
 

The port-layer log reports messages similar to the following:

Jul 30 2018 22:05:25.956Z ERROR op=300.5286: CommitHandler error on handle(3ff59a60855687480af2bd457bc7cda9) for 8c1ddfff634eeb2e7d225b978d60482aa9e04de9bbcb83243a794dfaba998823: ServerFaultCode: Permission to perform this operation was denied.
 

Note: The port-layer.log is located in /var/log/vic/port-layer.log on the VCH endpoint VM. The log can be accessed in the following ways:
Browser: https://VCH-IP-FQDN:2378/logs/port-layer.log
Shell: Enable the shell using "vic-machine debug", see link below. Once enabled connect a SSH session and run 'vi /var/log/vic/port-layer.log'. 

Cause

The user used in ether --user or --ops-user does not have sufficient permissions.

Resolution

Verify that the user used in --user is a full vSphere admin. If not then redeploy the VCH using a full vSphere admin for the --user option to ensure proper configuration. 

If the --ops-user was used in junction with --user then verify the ops user has all permissions listed in the documentation section for Manually Create a User Account for the Operations User. Alternatively can re-deploy the VCH adding the --ops-grant-perms option and the --user user will be used during the install to allocate the needed permissions to the ops-user. More details in the documentation in section Grant Any Necessary Permissions