VMware Cloud Foundation Workload Domain expansion fails at the Task "Configure new vCenters and ESXi hosts from workload domains in vRLI"

Products

• VMware Aria Suite
• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Workload Domain expansion fails at the task "Configure new vCenters and ESXi hosts from workload domains in vRLI".
• LogInsight VMs are not allowing logins, either by SSH (root) or any user through the UI (including admin).
• The root ( / ) filesystem is filling up or full (100% Use) on LogInsight VMs.
• Installation of Content Packs in Log Insight fails.

Environment

VMware Cloud Foundation 2.2.x
VMware vRealize Log Insight 4.3.x
VMware Cloud Foundation 2.3.x

Cause

The Log Insight log gathering function of the sos utility (on the SDDC Manager Controller VM) might not complete successfully leaving behind partial log bundles under /tmp/repo folder on the Log Insight VM.

Resolution

This issue is resolved in VCF 2.2.1 and later.

Workaround:
You can workaround this issue using the following steps:

1. Attempt to log in to the affected Log Insight node as the root user (ssh or console). If you are able to log in, proceed to step 2.  Otherwise see the procedure for booting off a Linux ISO in the Related Information section.
2. Issue the following command to remove the /tmp/repo directory:

rm -rf /tmp/repo

3. Issue the  df -h  command to validate that the root (/) file system is no longer full.
4. Repeat steps 1-3 for any other affected Log Insight nodes.

Additional Information

Note: If you are unable to log in as the root user (from Workaround step 1) and are certain you are using the correct root password, the root ( / ) filesystem is full on the affected Log Insight node.  You will therefore need to clean out the root ( / ) file system by booting from another Linux ISO and mounting the /dev/sda3 ( / ) filesystem from the booted ISO using the following procedure:

1. Acquire a Linux ISO to mount to the affected Log Insight node (VM).  In this example, we will be using an Arch Linux ISO as it is less than 1 GB (~600MB) in size.
2. Shut down the VM and edit the properties to mount the ISO you have uploaded to a local datastore accessible by the Log Insight VM(s).  Be sure to set the boot to BIOS option under "Boot Options" in order to reconfigure the BIOS to boot the CDROM first (instead of the First HDD).  This will make it easier to boot from the ISO as the VM BIOS screen/POST operation is incredibly quick.
3. Boot the affected Log Insight VM into the shell provided by the Linux ISO.   For Arch Linux, this is option 3 and will automatically start as the root user ( #  prompt).  If you are using a different distribution of Linux, you may need to become root with the command sudo su - before proceeding.
4. Issue the  fdisk -l  command to list all the available partitions, make sure /dev/sda3 is listed.  That will be the root ( / ) filesystem of the Log Insight VM.
5. Mount the root ( / ) partition using the following command:  mount /dev/sda3 /mnt
6. Issue the command df -h to verify /mnt  (/dev/sda3) is at 100% utilization (Note: the ISO you have booted off of will also show 100%, you want to pay attention to the /dev/sda partitions).
7. Change to the tmp filesystem on /dev/sda3 by issuing the command: cd /mnt/tmp
8. Issue the command: rm -rf repo  to remove the repo directory and all of it's contents. 
9. Verify the filesystem has has been freed up sufficiently by issuing the df -h command again.  The /mnt (/dev/sda3) partition should no longer be at 100% (it should be substantially less).  Change back to the ISO root directory and unmount the /mnt (/dev/sda3) partition with the commands:
   cd /
umount /mnt
    
10. Shut down the Log Insight VM and detach the ISO.  Follow the same process as in step 2 to change the boot order back to the original settings (HDD First).
11. Boot the Lo gInsight VM and verify functionality by logging in as root via ssh.

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.