After upgrading concourse from 7.4.5 to 7.9.1 worker VMS are falling

Products

• Concourse for VMware Tanzu

Issue/Introduction

Checking the worker log, you will see below error message -

/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables)

Cause

Checking the bosh vital output, we can see the worker VM is still using bosh Xenial stemcell -

worker/#####  failing        AZ1  ##.##.##.##  vm-####-###  2xlarge.disk  true    bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.131  Thu Jul 17 14:01:21 UTC 2025  0d 0h 9m 42s      0.07, 0.06, 0.03  -      1.0%  0.2%  0.0%  2% (640 MB)   0% (0 B)     48% (33i%)  1% (0i%)    -  

Resolution

Refer to Concourse v7.9.1+LTS-T release note, you need to bump dependencies for worker runtime to support Ubuntu Jammy Jellyfish. 

If your deployment is using guardian runtime, we strongly recommend to use Ubuntu Jammy stemcell on the Broadcom Support portal. Otherwise, a linux distribution with cgroups v1 and kernel version >= 5.15 has to be used to allow worker node to start up.

Customer can follow the documentation Updating Stemcells to update the Concourse deployment stemcell.