VIC Docker cp on running containers fails with connection reset by peer

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Using the "docker cp" command to copy large files to a container volume fails with messages similar to the following. 

docker -H VCH_FQDN:2376 --tls cp ./LargeFile.dat frosty_blackwell:/tmp/
Error response from daemon: Post http://127.0.0.1:2377/archive?deviceID=4ccee877cad61d3fe0098809c68b161415dcd2e34d912f53f2d5f1fc6186ce0d&filterSpec=eyJJbmNsdXNpb25zIjp7fSwiRXhjbHVzaW9ucyI6e30sIlJlYmFzZVBhdGgiOiJ0bXAiLCJTdHJpcFBhdGgiOiIifQ%3D%3D&store=container: read tcp 127.0.0.1:58198->127.0.0.1:2377: read: connection reset by peer

The docker-personality.log would show Portlayer had an error and the service is restarting at the time that the "connection reset by peer" was observed in the docker-client output. 
 

Sep 11 2018 21:56:58.005Z WARN  Event streaming from portlayer returned: errors.apiError{error:(*errors.errorString)(0xc4205c3db0), statusCode:500}
Sep 11 2018 21:56:58.005Z WARN  Exiting Events Monitor: errors.apiError{error:(*errors.errorString)(0xc4205c3db0), statusCode:500}
Sep 11 2018 21:56:58.005Z ERROR Restarting Portlayer event monitor due to error: Server error from portlayer: Unknown error from the interaction port layer: Get http://127.0.0.1:2377/events: dial tcp 127.0.0.1:2377: getsockopt: connection refused
Sep 11 2018 21:56:58.005Z WARN  Event streaming from portlayer returned: errors.apiError{error:(*errors.errorString)(0xc4209661f0), statusCode:500}
Sep 11 2018 21:56:58.005Z WARN  Exiting Events Monitor: errors.apiError{error:(*errors.errorString)(0xc4209661f0), statusCode:500}
Sep 11 2018 21:56:58.005Z ERROR Restarting Portlayer event monitor due to error: Server error from portlayer: Unknown error from the interaction port layer: Get http://127.0.0.1:2377/events: dial tcp 127.0.0.1:2377: getsockopt: connection refused

Note: The docker-personality.log is located in /var/log/vic/docker-personality.log on the VCH endpoint VM. The log can be accessed in the Browser at https://VCH-IP-FQDN:2378/logs/docker-personality.log

Environment

VMware vSphere Integrated Containers 1.4.x

Cause

Copying large files or directories to a running container result in an out of memory condition.

Resolution

Issue is currently being worked on. This KB can be subscribed to for future updates regarding the issue.

Workaround:
Perform the copy operations while the container is stopped. For example:

docker -H VCH_FQDN:2376 --tls stop MyContainerVM
docker -H VCH_FQDN:2376 --tls cp ./LargeFile.dat MyContainerVM:/tmp/
docker -H VCH_FQDN:2376 --tls start MyContainerVM

Additional Information

Supported Docker Commands - Container Commands
Docker cp on running containers fails with connection reset by peer #8285