hub-tas-agent job failing in Platform services tile due to disk space constraints

Products

• VMware Tanzu Platform Core

Issue/Introduction

• The hub-tas-agent job is not running in the Platform Services deployment.
• Running 'bosh vms' you will see the hub_tas_agent instance in 'failing' state:
  

  Deployment 'hub-tas-collector-<ID>'

  Instance                 Process State  AZ   IPs              VM CID                                 
ensemble_stitching/<ID>  running        az1  ###.###.###.###  vm-########-####-####-####-ac01f2ccbb20
hub_tas_agent/<ID>       failing        az1  ###.###.###.###  vm-########-####-####-####-94d5bf077a9e
logs-store/<ID>          running        az1  ###.###.###.###  vm-########-####-####-####-eefd7358ca40
telegraf/<ID>            running        az1  ###.###.###.###  vm-########-####-####-####-a958331b386f
  
  

• The Foundation reports 'Offline' in Tanzu Hub, all metrics for this attached foundation are not reporting.
• From a bosh ssh into the hub_tas_agent instance VM, running 'df -h' shows the /var/vcap/store filesystem full:
  
  hub_tas_agent/<ID>:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           392M   41M  352M  11% /run
/dev/sda2       4.8G  2.9G  1.8G  63% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sda1        48M  5.7M   43M  12% /boot/efi
tmpfs            16M   16K   16M   1% /var/vcap/bosh/settings
/dev/sdb2        28G  2.5G   24G  10% /var/vcap/data
tmpfs           100M  776K  100M   1% /var/vcap/data/jobs
tmpfs           100M     0  100M   0% /var/vcap/data/sensitive_blobs
tmpfs            16M  304K   16M   2% /var/vcap/data/sys/run
tmpfs            16M     0   16M   0% /var/vcap/bosh/canrestart
/dev/sdc1        30G   30G     0   100% /var/vcap/store
tmpfs           392M     0  392M   0% /run/user/1001
  
  

 

 

Cause

The persistent disk on the hub-tas-agent VM mainly stores SBOMS that have been downloaded from the Foundation that are sent up to Hub. A local cache is kept in order to prevent constant downloads from the Foundation. 

Resolution

Improvements have been made in the Tanzu Hub 10.4.1 patch release of Platform Services tile to optimize storage usage on the hub-tas-agent VM.

 

 

Workaround

If disk consumption still reaches 100% due to larger foundations, increase the Persistent Disk for the Hub TAS Agent job in the Platform Services tile Settings > Resource Config section in Opsman.