Unable to configure backups using VanDyke VShell on Windows (Handshake Failed) for Security Services Platform (SSP) and Security Services Platform Installer (SSPI)

Products

• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

 When attempting to configure or run automated backups from the Security Services Platform (SSP) or Security Services Platform Installer (SSPI) to a Windows-based server running VanDyke VShell SFTP, the backup fails with the following error:

"Failed to create SFTP client. Error: error creating sftp connection to server. sftp: sftpConnect: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"

Environment

SSP 5.1 and above

VanDyke VSHELL SFTP on Windows

Cause

VShell on Windows does not natively support the standard OpenSSH authorized_keys file format for public key authentication. While SSP and SSPI attempt to provision keys using standard methods, VShell requires individual .pub files stored in a specific directory structure for each user account.

Resolution

1. To resolve this, you must manually bridge the gap between the SSP/SSPI standard provisioning and VShell's proprietary key store by placing the public key in the correct VShell directory.

Create User Directory: Open PowerShell on the VShell server and create a directory named after the backup username within the VShell PublicKey folder:

New-Item -ItemType Directory -Force -Path "C:\Program Files\VanDyke Software\VShell\PublicKey\<username>"

(Replace <username> with the actual account used for backups, e.g., backup-user).

2. Extract and Save Public Key:

• Locate the public key provided by the Security Services Platform (SSP) or Security Services Platform Installer (SSPI) appliance (found in the .ssh/authorized_keys file on the appliance or provided during the backup configuration setup).
  

  OR

  - Retrieve SSPI public key from SSPI CLI from this location

  cat /home/usecop/.ssh/id_rsa.pub

  And SSP public key from cluster-api

  k exec -it <cluster-api-####> -n nsxi-platform -c cluster-api -- /bin/bash

  cat /home/vmware/.ssh/id_rsa.pub

• Create a new text file named id_rsa.pub (or key.pub) in the directory created in Step 1.
• Paste the public key content into this file.
• Note: If the authorized_keys contains multiple keys, VShell requires them to be saved as separate individual .pub files (e.g., ssp_id_rsa.pub, sspi_id_ed25519.pub).

3. Verify Backup: No restart of the VShell service is required. Re-run the backup from the SSP/SSPI interface to verify successful authentication.