VMware Cloud Foundation Upgrade fails due to LogInsight configuration issues

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:
VCF upgrade fails due to inability to connect to log insight services

• You can not connect to log insight through console or putty as the root account

Environment

VMware Cloud Foundation 2.3.x

Cause

The log insight servers are full and not allowing logins.  A large number of failed upgrades could fill up the /tmp directory causing issues on log insight server.

Resolution

This issue can occur and be seen when multiple upgrade attempts have occurred while attempting to upgrade VMware Cloud Foundation environment.

Follow this process for every appliance:

• Reboot appliance to single user mode
  1. Connect to the console of the VM through vCenter or ESXi host
  2. Reboot VM from vCenter or ESXi host using restart guest OS
     • Select the reset if restart guest OS does not work.
  3. During boot up, watch for the VMware SUSE Linux Enterprise splash screen
  4. Select the normal boot option
  5. add a space and the word single at the end of the line
  6. Finish booting
  7. You will see a request for the root password or Control-D to continue.  Enter the root password.
• Clean up space on root drive and specifically look in the /tmp folder
  1. Run df -h to find what file system is full
  2. Use du -sh * in the directory that is full to find what is using up all the space
  3. Remove the files in that directory (or possibly grow that disk)
• Reset the root password: clear /etc/security/opasswd file
• Reboot appliance