VMware Cloud Director backup failed due to incorrect ssh configuration

Products

• VMware Cloud Director

Issue/Introduction

• Creating backup in VMware Cloud Director(vcd) management interface fails with:
  
  Unable to create backup: A failure occurred while creating the backup on the primary.
  
  
• /opt/vmware/var/log/vcd/backup-remote.log reports below error:
  
  ERROR | Error running command: CompletedProcess(args='ssh postgres@<eth1 ip of primary cell> sudo /opt/vmware/appliance/bin/create-backup.sh', returncode=255, stdout='', stderr='Welcome to VMware Cloud Director VA\nPermission denied, please try again.\nPermission denied, please try again.\nReceived disconnect from <eth1 ip of primary cell> port 22:2: Too many authentication failures\nDisconnected from <eth1 ip of primary cell> port 22\n')
  
  
• If run below command from primary cell, vcd still prompts for password while it shouldn't require password to switch to postgres user:
  
  root@<vcd name> [ ~ ]# sudo -n -u postgres ssh postgres@<eth1 ip of primary cell>
Welcome to VMware Cloud Director VA
postgres@<eth1 ip of primary cell>'s password:

Environment

VMware Cloud Director 10.x

Cause

An incorrect "AllowUsers" setting was added to /etc/ssh/sshd_config which would deny ssh access to all other users not specified in the config file.
As a result postgres user fails to ssh into primary cell database to execute backup script which leads to manual backup failure.

Resolution

Remove the incorrect setting from ssh configuration:

1. Take a backup of the original config file:
   cp /etc/ssh/sshd_config /etc/ssh/sshd_config_bak
2. Remove or comment out the "AllowUsers" line from /etc/ssh/sshd_config with vi:
   vi /etc/ssh/sshd_config
3. Restart sshd service:
   service sshd restart

Additional Information

For instance, there's an "AllowUsers root nagios" setting added to /etc/ssh/sshd_config file which prevents postgres user from login with ssh and needs to be removed.