SDDC Manager Backup configuration failed with error - Invalid parameter: Validation failed for directory path

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

Unable to configure SDDC manager backup in site settings, receiving the below error:

Invalid parameter: Validation failed for directory path /storage/backup/sddcmanager/ on server 192.168.0.##. Please make sure backup directory is intact and sftp server has write permission on backup path.

You can find the error "End of IO Stream Read" in /var/log/vmware/vcf/operationsmanager/operationsmanager.log

YYYY-MM-DDTHH:MM DEBUG [vcf_om,20190509c0c4abc,0509] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,http-nio-127.0.0.1-7300-exec-4] Security config retrieved {"certificateValidationEnabled":true,"fipsMode":false}

YYYY-MM-DDTHH:MM ERROR [vcf_om,20190509c0c4abc,0509] [c.v.evo.sddc.common.util.SshUtil,http-nio-127.0.0.1-7300-exec-4] Unable to create jsch CLI session:

com.jcraft.jsch.JSchException: Session.connect: java.io.IOException: End of IO Stream Read

at com.jcraft.jsch.Session.connect(Session.java:565)

at com.vmware.evo.sddc.common.util.SshUtil.getSession(SshUtil.java:678)

at com.vmware.vcf.secure.ssh.SshExecuter.<init>(SshExecuter.java:98)

at com.vmware.vcf.secure.ssh.SshExecuterFactory.createSshExecuter(SshExecuterFactory.java:137)

at com.vmware.vcf.secure.ssh.SshExecuterFactory$$FastClassBySpringCGLIB$$d3e8b1e0.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

Environment

VMware SDDC Manager 5.X

Cause

The DEFAULT crypto-policy in RHEL9 systems does not allow SHA1.
Incompatibility between SSH Server and SDDC Manager.
This issue occurs in the following versions of the SFTP server. Newer versions may have the same issue.
```
Open SSH version: 8.7.1 
Guest OS: RHEL 9.2
```

Resolution

Currently, there is no resolution to this issue.

1. SSH Key Negotiation and OpenSSH Compatibility (VCF 4.x / 5.x)
Newer OpenSSH versions (8.8 and higher) or specific guest OS defaults (like RHEL 9) often disable older algorithms (e.g., SHA-1) that SDDC Manager expects.

KB 301484: This applies if your SFTP server is running RHEL 9.2+ or OpenSSH 8.7.1+. The RHEL 9 default crypto-policy does not allow SHA1, causing a JSchException: Session.connect: java.io.IOException: End of IO Stream Read.

Workaround: Add KexAlgorithms diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 to the bottom of /etc/ssh/sshd_config on the SFTP server and restart the SSH service.
Alternative: Temporarily enable SHA1 on the RHEL server using update-crypto-policies --set DEFAULT:SHA1 and reboot.

KB 367313 / KB 441702: Host key negotiation fails with reject HostKey or missing rsa-sha2. This is fixed in VCF 5.1.1.

Workaround: Manually register the SSH-RSA fingerprint via the SDDC Manager API.

Requirement: Ensure at least one of rsa-sha2-512 or rsa-sha2-256 and one of ecdsa-sha2-nistp256/384/521 is enabled on the SFTP server.

2. SSH Known Hosts Mismatch
KB 316034: Caused by a mismatch between the external SFTP server's current SSH keys and those stored in the SDDC Manager known_hosts file.
Resolution: You must update the SSH keys in the SDDC Manager's known_hosts file. Follow KB 316028 for the specific update procedure.

Workaround:

Add the following Key Algorithms to the bottom of the sshd_config file on the SFTP server.
```
KexAlgorithms diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
```
Restart the SSH service on the SFTP server.

Note:

When the operationsmanager.log shows the below error, then the workaround of updating the cryptographic policy to SHA1 on the affected RHEL9 server should be used:

DEBUG [vcf_om,67ed261d8521124815f852a87e027b2c,427e] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,pool-3-thread-9] Security config retrieved {"fipsMode":false}

ERROR [vcf_om,67ed261d8521124815f852a87e027b2c,427e] [c.v.evo.sddc.common.util.SshUtil,pool-3-thread-9] Unable to create jsch CLI session:
com.jcraft.jsch.JSchException: Session.connect: java.security.SignatureException: Bad signature length: got 404 but was expecting 384
at com.jcraft.jsch.Session.connect(Session.java:565)

Apply the SHA1 subpolicy to the DEFAULT cryptographic policy:
```
update-crypto-policies --set DEFAULT:SHA1
```
Reboot the system
Read more at Red Hat Documentation
Re-enabling SHA-1

Note:

After successfully configuring the backup, the SHA1 submodule can be removed again from the RHEL server as it weakens the security of the system. The backup process will still work.