Instructions to address CVE-2024-21626 for TKGI

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

CVE-2024-21626 was recently disclosed which impacts runc 1.1.11 and earlier. It has been found that TKGI is leveraging the impacted runc versions and is impacted by this CVE. The statement below, taken from the official CVE description, summarizes the potential attacks and impacts to the underlying platform.
 

In runc 1.1.11 and earlier, due to an internal file descriptor leak, an attacker could cause a newly spawned container process (from runc exec) to have a working directory in the host filesystem namespace, allowing for a container escape by giving access to the host filesystem ("attack 2"). The same attack could be used by a malicious image to allow a container process to gain access to the host filesystem through runc run ("attack 1"). Variants of attacks 1 and 2 could also be used to overwrite semi-arbitrary host binaries, allowing for complete container escapes ("attack 3a" and "attack 3b").

Resolution

To address this issue in the current supported versions of TKGI i.e. 1.16.x, 1.17.x and 1.18.x. The TKGI team has released intermediary patches that can address this issue. The instructions in this article will cover how you can identify if you are impacted by the CVE, apply fixes and confirm the platform has been patched.

Please note that this is an interim solution, and official TKGI patch releases will be released at a later date.

How to identify if you are impacted by this CVE?

All supported TKGI versions are impacted by this CVE. A quick way to confirm this is to target one of your clusters via bosh and query for the runc version on worker nodes. If the version is runc 1.1.11 and earlier the impact is confirmed.

export SD='service-instance_4e73e298-5801-4b5b-a271-2965cc01e08c'
bosh -d $SD ssh worker "sudo /var/vcap/data/packages/containerd/*/bin/runc --version" | grep runc

How to patch your TKGI Kubernetes clusters with the fix for this CVE?

The example in this KB is using the latest TKGI build (1.18.1) but the same steps can be followed for TKGI 1.16.x and TKGI 1.17.x as well. The steps will call out if there is a specific change needed for a particular TKGI version.

The outputs of the commands may differ in your environment as these steps were performed on a TKGI 1.18.1 environment.

Verify current runc version of your cluster

The version is 1.1.11 or lower confirms that the cluster is impacted by the CVE.

export SD='service-instance_4e73e298-5801-4b5b-a271-2965cc01e08c'

bosh -d $SD ssh worker "sudo /var/vcap/data/packages/containerd/*/bin/runc --version" | grep runc

worker/8f87f2f7-f8c6-41b8-94ec-d81deb3339d4: stdout | runc version 1.1.9
worker/6c3a9eb5-b069-4dbd-a490-21534d104b1c: stdout | runc version 1.1.9
worker/4be818f2-6bd4-4dd2-bb56-c852e8f3dc4b: stdout | runc version 1.1.9

Verify current containerd version of your cluster

The version lower than 1.6.28 confirms that the cluster is impacted by the CVE.

bosh ssh -d $SD worker "sudo /var/vcap/data/packages/containerd/*/bin/containerd-shim-runc-v2 -v" | grep Version

worker/8f87f2f7-f8c6-41b8-94ec-d81deb3339d4: stdout |   Version:  v1.6.24
worker/6c3a9eb5-b069-4dbd-a490-21534d104b1c: stdout |   Version:  v1.6.24
worker/4be818f2-6bd4-4dd2-bb56-c852e8f3dc4b: stdout |   Version:  v1.6.24

SSH to Opsmgr and Set environment variables for your TKGI version

SSH to Opsmgr and switch to the root user 

sudo su -

For TKGI 1.16

export PATCH_URL='https://storage.googleapis.com/shared-gss/kubo-1.16.0-build.103.fix-runc-CVE-for-1.16.x.tgz'
export PATCH_VERSION='1.16.0-build.103.fix-runc-CVE-for-1.16.x'
export TKGI_VERSION='1.16'

For TKGI 1.17

export PATCH_URL='https://storage.googleapis.com/shared-gss/kubo-1.17.0-build.54.fix-runc-CVE-for-1.17.x.tgz'
export PATCH_VERSION='1.17.0-build.54.fix-runc-CVE-for-1.17.x'
export TKGI_VERSION='1.17'

For TKGI 1.18

export PATCH_URL='https://storage.googleapis.com/shared-gss/kubo-1.18.0-build.26.fix-runc-CVE-for-1.18.x.tgz'
export PATCH_VERSION='1.18.0-build.26.fix-runc-CVE-for-1.18.x'
export TKGI_VERSION='1.18'

Download kubo Patch Release

Make sure the correct permissions are reflected after downloading the patch as shown in the output below

cd /var/tempest/releases
wget $PATCH_URL
chown tempest-web.tempest-web kubo-$PATCH_VERSION.tgz
shasum kubo-$PATCH_VERSION.tgz

root@opsman-local:/var/tempest/releases# ls -lrth | grep CVE
-rw-r--r-- 1 tempest-web tempest-web 461M Feb  8 11:33 kubo-1.18.0-build.26.fix-runc-CVE-for-1.18.x.tgz

You can verify the integrity of each downloaded patch release by comparing the shasum value

2cdef6fc6f48e6e217a53265a8b64b17b7c1e46d  kubo-1.18.0-build.26.fix-runc-CVE-for-1.18.x.tgz
a74c76a9be749fa0f5d5de606d24f78f3773534a  kubo-1.17.0-build.54.fix-runc-CVE-for-1.17.x.tgz
ad3cb2d1af26cb2ab943b63db5ab971c3746cf81  kubo-1.16.0-build.103.fix-runc-CVE-for-1.16.x.tgz

Find and Backup the TKGI tile manifest

grep -l "^product_version: $TKGI_VERSION" /var/tempest/workspaces/default/metadata/*

Take backup of the file returned by the above command. Example -

cp /var/tempest/workspaces/default/metadata/3b3dc80e51db.yml $HOME/.

Update TKGI tile manifest to include the patched kubo release

The following section of the config file from the last step under /var/tempest/workspaces/default/metadata/ needs modification. Please refer the relevant section for your TKGI version.

TKGI 1.18.x

Original Section
- name: kubo
  version: 1.18.0-build.25.1.18.x
  file: kubo-1.18.0-build.25.1.18.x-ubuntu-jammy-1.318.tgz
  exported_from:          <-- make sure remove this line
  - os: ubuntu-jammy      <-- make sure remove this line
    version: '1.318'             <-- make sure remove this line

 

Section After Update
- name: kubo
  version: 1.18.0-build.26.fix-runc-CVE-for-1.18.x
  file: kubo-1.18.0-build.26.fix-runc-CVE-for-1.18.x.tgz

Original Section
    service_deployment:
      releases:
        - name: kubo
          version: "1.18.0-build.25.1.18.x"
  
Section After Update
    service_deployment:
      releases:
        - name: kubo
          version: "1.18.0-build.26.fix-runc-CVE-for-1.18.x"

TKGI 1.17.x

Original Section
- name: kubo
  version: 1.17.0-build.52.1.17.x
  file: kubo-1.17.0-build.52.1.17.x-ubuntu-jammy-1.340.tgz
  exported_from:         <-- make sure remove this line
  - os: ubuntu-jammy     <-- make sure remove this line
    version: '1.340'     <-- make sure remove this line

Section After Update
- name: kubo
  version: 1.17.0-build.54.fix-runc-CVE-for-1.17.x
  file: kubo-1.17.0-build.54.fix-runc-CVE-for-1.17.x.tgz 

Original Section
    service_deployment:
      releases:
        - name: kubo
       version: "1.17.0-build.52.1.17.x"
 
Section After Update
    service_deployment:
      releases:
        - name: kubo
          version: "1.17.0-build.54.fix-runc-CVE-for-1.17.x"

TKGI 1.16.x
 

Original Section
- name: kubo
  version: 1.16.0-build.102.1.16.x
  file: kubo-1.16.0-build.102.1.16.x-ubuntu-jammy-1.340.tgz
  exported_from:             <-- make sure remove this line
  - os: ubuntu-jammy         <-- make sure remove this line
    version: '1.340'         <-- make sure remove this line
  
Section After Update
- name: kubo
  version: 1.16.0-build.103.fix-runc-CVE-for-1.16.x
  file: kubo-1.16.0-build.103.fix-runc-CVE-for-1.16.x.tgz
 
 
Original Section
    service_deployment:
      releases:
        - name: kubo
          version: "1.16.0-build.102.1.16.x"
  
Section After Update
    service_deployment:
      releases:
        - name: kubo
          version: "1.16.0-build.103.fix-runc-CVE-for-1.16.x"

Review Pending Changes

Login to the Opsmgr UI and click Review Pending Changes -> See Changes. The screenshots below highlight what to expect in terms of differences for your respective version.

TKGI 1.18.x



TKGI 1.17.x



TKGI 1.16.x


 

Apply Changes to TKGI Tile

• Make sure Upgrade All Clusters errand is not enabled in the TKGI tile.
• Run Apply Changes via Opsmgr

Upgrade Kubernetes Clusters

Run tkgi upgrade-cluster <cluster-name> on each cluster. This will push the new kubo-* release to the worker nodes and update the runc + containerd version. 

How to verify your TKGI Kubernetes clusters are patched?

Once patched your runc version should be 1.1.12 and containerd version 1.6.28.

Verify runc version
 

export SD='service-instance_4e73e298-5801-4b5b-a271-2965cc01e08c'
bosh ssh -d $SD worker "sudo /var/vcap/data/packages/containerd/*/bin/runc --version" | grep runc

worker/4be818f2-6bd4-4dd2-bb56-c852e8f3dc4b: stdout | runc version 1.1.12
worker/8f87f2f7-f8c6-41b8-94ec-d81deb3339d4: stdout | runc version 1.1.12
worker/6c3a9eb5-b069-4dbd-a490-21534d104b1c: stdout | runc version 1.1.12

Verify containerd version

bosh ssh -d $SD worker "sudo /var/vcap/data/packages/containerd/*/bin/containerd-shim-runc-v2 -v" | grep Version

worker/96bea77f-c3b5-480c-bee2-869570498989: stdout |   Version:  v1.6.28
worker/fdbb2774-811b-4e7f-bec3-31207df93076: stdout |   Version:  v1.6.28
worker/4ac7f623-2e26-4884-8091-66c38cab0740: stdout |   Version:  v1.6.28