Bringup or Workload Domain operations fail in VMware Cloud Foundation 2.2 on backup or restore of ESXi hosts sub-tasks

Products

• VMware Cloud Foundation

Issue/Introduction

• Bringup or Workload Domain operations fail in VMware Cloud Foundation 2.2 on backup or restore of ESXi hosts sub-tasks:
• Specific sub-tasks that might fail are:

Backup Bootbank of Hosts
Initial Backup of the ESXi Servers
Restore Hosts

• Messages similar to the following are present in the /opt/vmware/bringup/logs/evosddc-bringup-debug.log file (during Bringup failure):

YYYY-MM-DDTHH:MM:SS [threadPoolExecutor-2] DEBUG [com.vmware.evo.sddc.common.util.SshUtil] <7e8c3fc0-XXXXX-11e7-a8cb-XXXXXXX> End of execution of command [ls], Status: 1
Output: 
Error output: Password change required but no TTY available.

Command timed out: false
YYYY-MM-DDTHH:MM:SS [threadPoolExecutor-2] INFO  [c.v.e.s.common.util.command.SshCommandExecuter] <7e8c3fc0-XXXXX-11e7-a8cb-XXXXXXX> Command 'ls' output:
Exit Code 1: 
Error output: Password change required but no TTY available.
YYYY-MM-DDTHH:MM:SS [threadPoolExecutor-2] ERROR [c.v.e.sddc.orchestrator.model.error.ErrorFactory]  [3ICRPP] HOST_NOT_REACHABLE Cannot connect to host with IP address 192.xxx.xxx.xx
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Cannot connect to host with IP address 192.xxx.xxx.xx

• Messages similar to the following are present in the /home/vrack/vrm/logs/vrack-vrm-debug.log file (during Workload Domain creation/deletion):

YYYY-MM-DDTHH:MM:SS [pool-26-thread-2] DEBUG [com.vmware.vrack.vrm.iaas.tools.service.EsxiBootbankBackupServiceImpl] Removing temp directory /home/vrack/vrm/temp/esxibackuplaststateba>ckup4896147657359965024
YYYY-MM-DDTHH:MM:SS [pool-26-thread-2] ERROR [com.vmware.vrack.vrm.workflow.tasks.vsphere.BackupHostTask] Could not backup configuration of host 192.xxx.xxx.xx
com.vmware.vrack.vrm.core.error.InternalServerErrorException: Error while taking /backup/hostbackup/R1N3/statebackuplatest backup of ESXi folder to LCM backup storage.

Caused by: com.jcraft.jsch.JSchException: java.io.IOException: Pipe closed

Resolution

Issue: A known issue in VMware Cloud Foundation 2.2 causes password expiration for the backupuser and vrack users on the SDDC Manager Utility and Controller VMs, respectively, leading to workflow failures.

Resolution: This issue is fixed in VMware Cloud Foundation 2.2.1.

Workaround:

1. SDDC Manager Utility VM:

   • Login: Log in to the SDDC Manager Utility VM as the root user.
   • Obtain root password: You can retrieve the root password by running the command /home/vrack/bin/lookup-passwords on the SDDC Manager Controller VM.
   • Modify backupuser password policy: Execute the following command to set a 365-day password expiry for the backupuser:
     bash chage -M 365 backupuser

2. SDDC Manager Controller VM:

   • Login: Log in to the SDDC Manager Controller VM as the root user.
   • Modify vrack password policy: Execute the following command to disable password expiration for the vrack user:
     bash chage -M 9999 vrack

3. Retry Workflow: After modifying the password policies, go to the SDDC Manager UI and click the "Retry" button on the failed workflow.

Note:

• The provided workaround should only be considered a temporary solution until upgrading to VMware Cloud Foundation 2.2.1.
• Carefully follow the instructions and ensure you have appropriate privileges to access the SDDC Manager VMs and execute the commands.

Recommended Action: Upgrade to VMware Cloud Foundation 2.2.1 as soon as possible to permanently resolve this issue.