Creating a hostprocess app in a TKGI Windows cluster fails with the error "hostProcess containers are disallowed by cluster policy"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Deploying a Windows application in a TKGI Windows cluster fails with the following error:

$ kubectl -n windows apply -f application.yaml
The Deployment "hostprocess" is invalid: spec.template.spec: Forbidden: hostProcess containers are disallowed by cluster policy

 

Environment

TKGI v1.22.0

Cause

The hostProcess feature needs the kube-apiserver to allow privileged containers.  The allow-privileged setting in kube-apiserver is not user-configurable in the plans for Windows clusters and is disabled (set to false) by default.

Resolution

Caution:  Allowing pods to run in privileged mode will give the container elevated permissions that allow it to perform operations that would be restricted otherwise.  The privileged container will have the same level of access to the Windows node as the administrator.  Some recommendations around this are:

• Create a specific account with the required permissions on the Windows node. This might mean adding the account during stemcell building, and designing and creating the container to use that account.
• If hostProcess is still needed, then create a validating webhook to only allow specific cluster users to be able to create hostProcess applications.

---

 

The "allow-privileged=true" parameter needs to be added to the kube-apiserver flags in the /var/vcap/jobs/kube-apiserver/config/bpm.yml file.

As a workaround, to make the changes to the bpm.yml file persistent, the following steps can be used.

 

1. Upload the bosh os-conf release. 

bosh upload-release --sha1 d20772d8ce6e781ceb13cac7df5950bfa4330ba1 \
  "https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=23.0.0"

 

2. Confirm if the release was uploaded.

bosh releases | grep os-conf
os-conf                        	23.0.0              	a1905d6

 

3. Create a runtime config that will create a script in the master instances. The script will be executed every time the master instance is started by bosh.  When executed, the script will add the necessary setting to the bpm.yml file, and restart the kube-apiserver job.  Make sure to replace the values in .addons.exclude.deployments with the appropriate unique deployment names.

cat <<'EOFA' > runtime.yml
releases:
- name: "os-conf"
  version: "23.0.0"
addons:
- name: enable-allow-privileged-setting
  exclude:
    deployments: [pivotal-container-service-xxx,cf-xxx]
  include:
    instance_groups: [master]
  jobs:
  - name: pre-start-script
    release: os-conf
    properties:
      script: |-
        #!/bin/bash
        sed -i '/--tls-cipher-suites/a \ \ - "--allow-privileged=true"' /var/vcap/jobs/kube-apiserver/config/bpm.yml
        cat /var/vcap/jobs/kube-apiserver/config/bpm.yml | grep allow-privileged
        monit restart kube-apiserver
        echo "done"
EOFA

 

4. Update the runtime config.

bosh update-runtime-config runtime.yml

 

5.Verify the runtime-config was updated.

bosh runtime-config

 

6. Upgrade the cluster

tkgi upgrade-cluster windowscluster