After upgrading ImagePullBack issue observed for the Airgapped Environment

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

• Post upgrade, TCA 3.3 failed to pull the container images after the reboot

Environment

TCP: 5.0.2

TCA: 3.3

Cause

After upgrading to version 3.3.0, a cleanup script removes old TCA images. However, due to a bug in the script, some off-the-shelf (OTS) images (e.g., Kafka, Postgres, Istio) that remained unchanged between versions 3.2.0.1 and 3.3.0 are also being inadvertently deleted.

As a result, if any pods using these images are recreated, they fail to start and enter an ImagePullBackOff state because the corresponding images are no longer available locally.

Pods can be recreated in several scenarios, including:

• Restart of any service triggered via the Platform Manager UI.
• Pod crashes due to OOM kills or other failures.
• VM reboot.
• Manual deletion of the pod to force a restart, etc.

 

Resolution

Resolved in TCA 3.3.0.1

 

Workaround:

To fix the issue, download the patch manually and apply it to both TCA-M and TCA-CP appliances.

1. Download the attached recover-missing-images-tca-3-3-0-patch.tar.gz file from the KB.
   
   
2. Copy the "recover-missing-images-tca-3-3-0-patch.tar.gz" file to all 3.3.0 TCA-M and TCA-CP VMs (Example copy/scp the file to TCA-M > /tmp )
   
   
3. Switch to root user and execute the following commands:
   

   tar -xvzf recover-missing-images-tca-3-3-0-patch.tar.gz
   
   bash recover-missing-images-tca-3-3-0-patch/recover-missing-images-tca-3-3-0.sh

4. After applying the above patch, the impacted images will be restored, the pods will be back, and Image pull will work as expected.

Additional Information

NOTE : MD5 | SHA256 details of recover-missing-images-tca-3-3-0-patch.tar.gz

MD5: 1b88f048e2645c9978502e9a441757d0

SHA256: 85a34abfe7ffd038ad312338fa5c685105cb89331d18deeafb14d8a651a819ea