CSE cluster creation in VMware Cloud Director remains in "pending" state

Products

• VMware Cloud Director

Issue/Introduction

When creating a new Kubernetes cluster in Container Service Extension(CSE), the status remains as Pending.

 

Environment

VMware Cloud Director 10.5.x

Container Service Extension: 4.x

Cause

100% capacity used for /dev/sda3 on the CSE VM.

Resolution

To resolve the issue:

1. Take a snapshot of the CSE VM.
2. Reboot the CSE VM from vCenter.
3. After the vm starts, press the e key to enter the GNU GRUB Edit Menu.
4. Locate the line that begins with the word Linux.
5. Append these entries to the end of the line -

rw init=/bin/bash

6. Press F10 to continue booting.

7. Run the command: 

mount -o remount,rw / 

8. Navigate to the /var/log/audit folder with this command:

cd /var/log/audit

9. Run this command to verify the issue is with the audit.log file being too large (a few GBs):

ls -lh

For example:

ls -lh

total 3.5G

-rw------- 1 root root 3.5G Feb 3 16:55 audit.log
-rw------- 1 root root 445K Apr 8 2016 audit.log-20160408.bz2
-rw------- 1 root root 447K Apr 9 2016 audit.log-20160409.bz2
 

10. Truncate (clean the content without deleting the file) the audit.log file with this command:

truncate -s 0 audit.log*

11. Unmount the filesystem by running this command

umount / 

12. Reboot the CSE VM by running this command:

reboot -f

   13. Wait for the CSE service to start.

14. Create the CSE cluster from VCD UI.

Additional Information

Run the following command to identify large files in the root partition.

du -ahx / | sort -rh | head -20

The above example will list the first 20 entries consuming maximum space.