VCFA SFTP Configuration failure on VCF Fleet Management – LCMVMSP10035 Handshake Error

Products

• VCF Operations
• VCF Automation

Issue/Introduction

Configuring SFTP in VCF Fleet Management for VCFA fails with error LCMVMSP10035: "failed to connect to SSH server: ssh: handshake failed".

Although the same SFTP server works successfully with SDDC, NSX, Fleet Management reports the following full error message during the synthetic check:
"Failed to connect to SSH server: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain."

Environment

VCF Operations 9.0
VCF Operations 9.0.1
VCF Automation 9.0

Cause

The issue is caused by unsupported authentication methods on the SFTP server.
The SFTP server is configured to allow only keyboard-interactive authentication, while PasswordAuthentication is not enabled.

Synthetic-Checker requires password-based authentication, resulting in failed SSH handshake attempts.

Resolution

Broadcom is aware of this issue, and a permanent fix is planned for the VCF 9.0.2 release.

Workaround:

Option 1:

• Update the SFTP server configuration to enable PasswordAuthentication=true instead of relying solely on keyboard-interactive authentication.

Option 2:

• Temporarily disable the synthetic-checker on the VCFA node using the command:
  kubectl patch rt $(kubectl get rt -n vmsp-platform | grep synthetic-checker | awk '{print $1}') -n vmsp-platform --type='json' -p='[{"op": "replace", "path": "/spec/helm/values/sftpCheck/enable", "value": false}]' 

• Retry the failed SFTP task from Fleet Management or initiate a new configuration task. This time, the SFTP configuration will complete successfully.

NOTE:

Post performing Option-2 the Fleet Management UI may continue to display the SFTP status as ‘FAILED’ with the message ‘SFTP configuration failed on the cluster. Check the cluster logs for more details.’
This is expected and does not impact functionality. Scheduled backups for VCFA—including both incremental and full backups—will continue to run successfully.

Additional Information

How to verify the existing cluster configuration:

• Login on the (VCFA) cluster node (SSH) with the `vmware-system-user` user
• Change the user login to root by running sudo -i
• Export the KUBECONFIG env var with
  export KUBECONFIG=/etc/kubernetes/admin.conf

• List the current SFTP configuration
  kubectl get pd vmsp-platform -n vmsp-platform -ojsonpath="{.spec.values.sftp}"
kubectl get secret sftp-password-secret -n vmsp-platform -ojson | jq -r '.data | map_values(@base64d)'
• Validate SFTP connectivity from the node
  curl --connect-timeout 15 -slu "$USER:$PASSWORD" sftp:/$HOST:$PORT/
• Validate SFTP folder permissions
  touch testfile && curl -su "$USER:$PASSWORD" sftp://$HOST:$PORT/$DIRECTORY/testfile -T testfile 
• Check the synthetic-checker logs
  kubectl logs -c synthetic-checker -n vmsp-platform synthetic-checker-****-***