CNF Instantiation Failure Due to Full Disk on Telco Cloud Automation

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

Symptoms:
The following error may be encountered during CNF instantiation:
Helm API failed: HARBOR-CHART-URL is not a valid chart repository or cannot be reached: HARBOR-CHART-URL

Here HARBOR-CHART-URL is what is passed as the Repo URL during CNF instantiation.

Logging into the TCA-CP appliance via SSH and running df -kh shows the / partition is full.

Environment

VMware Telco Cloud Automation 2.0
VMware Telco Cloud Automation 1.x

Cause

The internal journal logs on the TCA-CP appliance do not roll over as expected, hus filling the / (root) partition..

This is a known issue and will be addressed in a future release

Resolution

1. Take a backup before patching the appliance VMs
   1. Take a Tech Support Bundle of the TCA and TCA-CP appliances to store current logs
   2. SSH to the TCA-CP and TCA appliance VMs and switch to root

 

2. Download and use the attached script to clean up old files and patch journald, log rotation fix:

# bash patch_journald_logrotate.sh 

Existing Diskspace
==================
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       5.6G  4.7G  658M  88% /

- Delete logs from previous upgrade

- Journalctl Vacuum
Deleted archived journal /var/log/journal/ba565ce64b5d48b99540fa417e0a96a5/system@35ace629b0e04d4e85d4345ba93d894b-0000000001551ddc-0005d851e547fde2.journal (20.0M).
Deleted archived journal /var/log/journal/ba565ce64b5d48b99540fa417e0a96a5/system@35ace629b0e04d4e85d4345ba93d894b-000000000156038c-0005d851e592bb36.journal (20.0M).
Deleted archived journal /var/log/journal/ba565ce64b5d48b99540fa417e0a96a5/system@35ace629b0e04d4e85d4345ba93d894b-000000000156e952-0005d851e5dbcff0.journal (20.0M).
Deleted archived journal /var/log/journal/ba565ce64b5d48b99540fa417e0a96a5/system@35ace629b0e04d4e85d4345ba93d894b-000000000157cf12-0005d851e626a810.journal (20.0M).
Deleted archived journal /var/log/journal/ba565ce64b5d48b99540fa417e0a96a5/system@35ace629b0e04d4e85d4345ba93d894b-000000000158b4d8-0005d851e6728a5d.journal (20.0M).
Vacuuming done, freed 100.1M of archived journals on disk.

- Delete Journal entry from logrotate

- Clean logrotate files

- Reapply logrotate config

- Fix journald config

New Diskspace
==================
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       5.6G  4.0G  1.4G  75% /

If the diskspace issue still persists, please check other folders for any stale files.