Configuring backup on VCF Operations is failing with File copy error: Failed to create file system for destination

Products

• VCF Operations
• VMware Cloud Foundation

Issue/Introduction

• Configuring Windows based sftp backup for VMSP component using VCF Operations UI fails with below

File copy error: Failed to create file system for destination "sftp:/<backup_directory>/": NewFs: couldn't connect SSH: ssh: handshake failed: knownhosts: key is unknown;

• The same Windows based backup server can be used for other components (SDDC, vCenter,NSX etc)

Environment

VCF 9.1

Cause

VMware Management Services Platform (VMSP) backup requires remote-exec permission to transfer data on the target sftp server. In case if the same is not available, the backups are expected to fail.

Resolution

Broadcom engineering is aware of the issue and will be fixed in future release.

Workaround:

As a workaround, install OpenSSH or Rclone on the backup server and reconfigure backup from VCF Operations UI

Configuring OpenSSH Server on Windows SFTP server

Configuring Rclone on Windows SFTP Server

Configuring OpenSSH Server on Windows

1. Install OpenSSH Server

1. 1. Open PowerShell as an Administrator
   2. Run the following command to check if OpenSSH Server is already available for installation:
      
      Get-WindowsCapability -Online | Where-Object Name -like 'OpenSSH.Server*'
      
      
   3. Install OpenSSH Server by running:
      
      Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
      (If successful, the output will return State = Installed.)
      
      
2. Configure the SSH Server Service

3. 1. Configure the service to start automatically and run
      
      Set-Service -Name sshd -StartupType 'Automatic'
Start-Service sshd
      
      
   2. Verify that the service is running
      
      Get-Service sshd

4. Configure the Windows Firewall and enable the built-in firewall rule for OpenSSH Server
   
   Set-NetFirewallRule -Name "OpenSSH-Server-In-TCP" -Enabled True
   
   
5. Configure SFTP Subsystem and Shell
   
   
   1. Open the configuration file using Notepad as Administrator
      
      notepad "$env:ProgramData\ssh\sshd_config"
      
      
   2. Scroll down and verify/ensure that the SFTP subsystem line is uncommented and points to sftp-server.exe
      
      Subsystem sftp sftp-server.exe
      
      
   3. Map user/group to only use SFTP by adding the below to the very bottom of the file
      
      For mapping user
      
      Match User <sftpuser>
    ChrootDirectory C:\SFTPRoot
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no
      
      For mapping group
      
      Match Group <sftp_usergroup>
    ChrootDirectory C:\SFTPRoot
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no
      
      
   4. Save and close the file
   5. To allow the user to upload and manage files, create a subfolder inside the root directory and grant write access to that subfolder
      
      # Create the root and upload subdirectories
      New-Item -ItemType Directory -Path "C:\SFTPRoot\uploads" -Force
      
      #Grant full control on the uploads subfolder to the SFTP user
      icacls "C:\SFTPRoot\uploads" /grant "sftpuser:(OI)(CI)F"
      
      
   6. Restart the SSH service to apply changes

Restart-Service sshd

Configuring Rclone on Windows SFTP server

1. Install Rclone
   
   
   1. Download the latest version of Rclone to a desired folder from https://downloads.rclone.org/
   2. Extract the rclone-current-windows-amd64.zip to a folder
   3. Open PowerShell as Administrator.
   4. Run the following commands to add Rclone to your System Path

# Add C:\rclone to System PATH variable so you can run it anywhere

$envPath = [Environment]::GetEnvironmentVariable("Path", "Machine")
if ($envPath -notlike "*C:\rclone*") {
    [Environment]::SetEnvironmentVariable("Path", "$envPath;C:\rclone", "Machine")
    $env:Path += ";C:\rclone"
}

Note: change the envPath as per the folder of rclone

# Verify installation

rclone version

# Update the  known_hosts_file 

rclone config update sftp-backup known_hosts_file "none"

2. Configure SFTP server and user

# For Password Authentication

rclone config create sftp-backup sftp host "<sftp IP/FQDN>" user "<sftpuser>" port 22 pass "YourPasswordHere"

OR

# For SSH Key Authentication

rclone config create sftp-backup sftp host "<sftp IP/FQDN>" user "<sftpuser>" port 22 key_file "C:\<path to id_rsa>"

3. Test a Backup Command

# Sync local folder 'C:\backup_folder' to remote SFTP folder '/remote/backup/path'

rclone sync <partition>:\<backup_folder> sftp-backup:/remote/backup/path --dry-run -P