Applications crash after upgrading due to "no privileged_image_plugin provided"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

After upgrading to PCF 1.12 or 2.0 from 1.11 or below, app containers are crashing.

The application was previously running in privileged-mode.

grootfs is enabled in Elastic Runtime or Isolation Segment tile.

Error Message:

App crashes with "failed to initialize container":

{Tue Feb 06 2018 08:43:38.218 [API/3] OUT App instance exited with guid 6c7a830d-21b2-452d-88b0-3ed78ac384b5 payload: {"instance"=>"7480c395-1920-4407-5088-4829", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"failed to initialize container", "crash_count"=>29, "crash_timestamp"=>1517928218153934394, "version"=>"cd84fde1-cfb7-45c4-b57f-357955fa0bc1"}

Diego cell rep logs report "no privileged_image_plugin provided"

"timestamp":"1518012638.901621580","source":"rep","message":"rep.executing-container-operation.ordinary-lrp-processor.process-reserved-container.run-container.containerstore-create.node-create.failed-to-create-container","log_level":2,"data":{"container-guid":"91cca2b7-5412-4658-4cf4-7988","container-state":"reserved","error":"creating create command: no privileged_image_plugin provided","guid":"91cca2b7-5412-4658-4cf4-7988","lrp-instance-key":{"instance_guid":"91cca2b7-5412-4658-4cf4-7988","cell_id":"ccf0e967-ebb9-43b9-a4e9-f8525f06dea4"},"lrp-key":{"process_guid":"92a8dca4-b615-43a4-b5aa-1d3879c58576-d0ac1167-65ab-4af5-ae51-8bcd85225c5b","index":0,"domain":"cf-apps"},"session":"7446.1.1.3.2.1"}}

Environment

Cause

grootfs which was introduced in 1.12 does not support privileged containers. Formerly, CF ran applications based on Docker images in unprivileged containers, and the buildpack-based applications and staging tasks in privileged containers. In new versions, all applications should run as unprivileged.

 

Resolution

The workaround is to restart the application. This will flip the container mode so that app container runs as unprivileged. To do this, run command:

# cf restart app-name

In order to identify apps running as privileged, you can bosh ssh to a Diego VM and run the command:

# cfdot desired-lrps | jq -r 'select(.privileged==true) | .process_guid' | cut -c-36`

This will get the guids of the affected apps. You can run command `cf /v2/apps/<GUID>` to get the application name associated with this GUID.

This issue is resolved in version 1.12.15 and 2.0.6 where privileged containers are supported again:

[Bug Fix] Enable privileged containers to support upgrading from ERT 1.11 with apps that specify privileged containers.

One further workaround option is to disable grootfs. 

There is an option “Enable the GrootFS container image plugin for Garden RunC” in ERT > Application Containers which can be used to disable grootfs. Unselect this option and click "Apply Changes" in OpsManager to disable grootfs from running on Diego cells.

We discourage this option of disabling grootfs unless absolutely needed.  Grootfs will become the only option in a future release, when the legacy option is removed.  If you are seeing issues with Grootfs and need to disable it, please make sure you contact Pivotal Support so we can log the issue with product engineering.